{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# XGBoost Parameter Tuning for Rental_listing Dataset-2\n",
    "  调整参数"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 第二步：调整树的参数：max_depth & min_child_weight¶\n",
    "(粗调，参数的步长为2；下一步是在粗调最佳参数周围，将步长降为1，进行精细调整)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 149,
   "metadata": {},
   "outputs": [],
   "source": [
    "#导入需要的包\n",
    "from xgboost import XGBClassifier\n",
    "import xgboost as xgb\n",
    "\n",
    "import pandas as pd \n",
    "import numpy as np\n",
    "from sklearn.metrics import accuracy_score\n",
    "from sklearn.model_selection import GridSearchCV\n",
    "from sklearn.model_selection import StratifiedKFold\n",
    "\n",
    "from sklearn.metrics import log_loss\n",
    "\n",
    "from matplotlib import pyplot\n",
    "import seaborn as sns\n",
    "%matplotlib inline"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>bathrooms</th>\n",
       "      <th>bedrooms</th>\n",
       "      <th>price</th>\n",
       "      <th>price_bathrooms</th>\n",
       "      <th>price_bedrooms</th>\n",
       "      <th>room_diff</th>\n",
       "      <th>room_num</th>\n",
       "      <th>Year</th>\n",
       "      <th>Month</th>\n",
       "      <th>Day</th>\n",
       "      <th>...</th>\n",
       "      <th>walk</th>\n",
       "      <th>walls</th>\n",
       "      <th>war</th>\n",
       "      <th>washer</th>\n",
       "      <th>water</th>\n",
       "      <th>wheelchair</th>\n",
       "      <th>wifi</th>\n",
       "      <th>windows</th>\n",
       "      <th>work</th>\n",
       "      <th>interest_level</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>1.5</td>\n",
       "      <td>3</td>\n",
       "      <td>3000</td>\n",
       "      <td>1200.0</td>\n",
       "      <td>750.000000</td>\n",
       "      <td>-1.5</td>\n",
       "      <td>4.5</td>\n",
       "      <td>2016</td>\n",
       "      <td>6</td>\n",
       "      <td>24</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>1.0</td>\n",
       "      <td>2</td>\n",
       "      <td>5465</td>\n",
       "      <td>2732.5</td>\n",
       "      <td>1821.666667</td>\n",
       "      <td>-1.0</td>\n",
       "      <td>3.0</td>\n",
       "      <td>2016</td>\n",
       "      <td>6</td>\n",
       "      <td>12</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>1.0</td>\n",
       "      <td>1</td>\n",
       "      <td>2850</td>\n",
       "      <td>1425.0</td>\n",
       "      <td>1425.000000</td>\n",
       "      <td>0.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>2016</td>\n",
       "      <td>4</td>\n",
       "      <td>17</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>1.0</td>\n",
       "      <td>1</td>\n",
       "      <td>3275</td>\n",
       "      <td>1637.5</td>\n",
       "      <td>1637.500000</td>\n",
       "      <td>0.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>2016</td>\n",
       "      <td>4</td>\n",
       "      <td>18</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>1.0</td>\n",
       "      <td>4</td>\n",
       "      <td>3350</td>\n",
       "      <td>1675.0</td>\n",
       "      <td>670.000000</td>\n",
       "      <td>-3.0</td>\n",
       "      <td>5.0</td>\n",
       "      <td>2016</td>\n",
       "      <td>4</td>\n",
       "      <td>28</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>5 rows × 228 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "   bathrooms  bedrooms  price  price_bathrooms  price_bedrooms  room_diff  \\\n",
       "0        1.5         3   3000           1200.0      750.000000       -1.5   \n",
       "1        1.0         2   5465           2732.5     1821.666667       -1.0   \n",
       "2        1.0         1   2850           1425.0     1425.000000        0.0   \n",
       "3        1.0         1   3275           1637.5     1637.500000        0.0   \n",
       "4        1.0         4   3350           1675.0      670.000000       -3.0   \n",
       "\n",
       "   room_num  Year  Month  Day       ...        walk  walls  war  washer  \\\n",
       "0       4.5  2016      6   24       ...           0      0    0       0   \n",
       "1       3.0  2016      6   12       ...           0      0    0       0   \n",
       "2       2.0  2016      4   17       ...           0      0    0       0   \n",
       "3       2.0  2016      4   18       ...           0      0    0       0   \n",
       "4       5.0  2016      4   28       ...           0      0    1       0   \n",
       "\n",
       "   water  wheelchair  wifi  windows  work  interest_level  \n",
       "0      0           0     0        0     0               1  \n",
       "1      0           0     0        0     0               2  \n",
       "2      0           0     0        0     0               0  \n",
       "3      0           0     0        0     0               2  \n",
       "4      0           0     0        0     0               2  \n",
       "\n",
       "[5 rows x 228 columns]"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# 读取数据\n",
    "#dpath = './data/'\n",
    "train = pd.read_csv(\"RentListingInquries_FE_train.csv\")\n",
    "test= pd.read_csv(\"RentListingInquries_FE_test.csv\")\n",
    "train.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAZUAAAELCAYAAAARNxsIAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMS4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvNQv5yAAAGx5JREFUeJzt3X2wHXWd5/H3h/AgPibIhcokYYKaGUUdo14hypSj6EDAHRMpUChHIkNtxAKF0rUEd5aMPMyM4ygrM8pupogkrkOIiBI1GDMMyII8JGB4CJHlioxcyZJgeNQVivDZP/p35eTm5N7OpU9OTu7nVdV1ur/96z7fwy340v379a9lm4iIiCbs0e0EIiJi95GiEhERjUlRiYiIxqSoREREY1JUIiKiMSkqERHRmBSViIhoTIpKREQ0JkUlIiIas2e3E9jZ9t9/f0+fPr3baURE9JTbbrvtEdt9o7Ubd0Vl+vTprFmzpttpRET0FEn/Uaddbn9FRERjOlZUJL1I0q2S7pC0TtLnS/xSSb+QtLYsM0tcki6SNCDpTklvaTnXPEn3lWVeS/ytku4qx1wkSZ36PRERMbpO3v56GjjC9lOS9gJukHR12fcZ21cMa380MKMshwEXA4dJ2g9YAPQDBm6TtNz2o6XNfOBmYAUwG7iaiIjoio5dqbjyVNncqywjzbM/B1hSjrsZmChpMnAUsMr25lJIVgGzy76X277J1fz9S4C5nfo9ERExuo72qUiaIGktsJGqMNxSdl1QbnFdKGmfEpsCPNhy+GCJjRQfbBNvl8d8SWskrdm0adML/l0REdFeR4uK7S22ZwJTgUMlvQE4G3gt8DZgP+CzpXm7/hCPId4uj4W2+2339/WNOiIuIiLGaKeM/rL9GHAdMNv2hnKL62ng68ChpdkgMK3lsKnAQ6PEp7aJR0REl3Ry9FefpIllfV/gvcDPSl8IZaTWXODucshy4KQyCmwW8LjtDcBK4EhJkyRNAo4EVpZ9T0qaVc51EnBVp35PRESMrpOjvyYDiyVNoCpey2x/X9K/S+qjun21Fji1tF8BHAMMAL8FTgawvVnSecDq0u5c25vL+seBS4F9qUZ9ZeRXREQXqRo4NX709/c7T9RH7NoO/6fDu53Cbu/GT9y4Q+0l3Wa7f7R2eaI+IiIak6ISERGNSVGJiIjGpKhERERjUlQiIqIxKSoREdGYFJWIiGhMikpERDQmRSUiIhqTohIREY1JUYmIiMakqERERGNSVCIiojEpKhER0ZgUlYiIaEyKSkRENCZFJSIiGpOiEhERjUlRiYiIxqSoREREY1JUIiKiMR0rKpJeJOlWSXdIWifp8yV+sKRbJN0n6XJJe5f4PmV7oOyf3nKus0v8XklHtcRnl9iApLM69VsiIqKeTl6pPA0cYftNwExgtqRZwBeAC23PAB4FTintTwEetf0a4MLSDkmHACcArwdmA1+TNEHSBOCrwNHAIcCJpW1ERHRJx4qKK0+Vzb3KYuAI4IoSXwzMLetzyjZl/3skqcSX2n7a9i+AAeDQsgzYvt/2M8DS0jYiIrqko30q5YpiLbARWAX8HHjM9rOlySAwpaxPAR4EKPsfB17ZGh92zPbiERHRJR0tKra32J4JTKW6snhdu2blU9vZt6PxbUiaL2mNpDWbNm0aPfGIiBiTnTL6y/ZjwHXALGCipD3LrqnAQ2V9EJgGUPa/AtjcGh92zPbi7b5/oe1+2/19fX1N/KSIiGijk6O/+iRNLOv7Au8F1gPXAseVZvOAq8r68rJN2f/vtl3iJ5TRYQcDM4BbgdXAjDKabG+qzvzlnfo9ERExuj1HbzJmk4HFZZTWHsAy29+XdA+wVNL5wE+BS0r7S4BvSBqgukI5AcD2OknLgHuAZ4HTbG8BkHQ6sBKYACyyva6DvyciIkbRsaJi+07gzW3i91P1rwyP/w44fjvnugC4oE18BbDiBScbERGNyBP1ERHRmBSViIhoTIpKREQ0JkUlIiIak6ISERGNSVGJiIjGpKhERERjUlQiIqIxKSoREdGYFJWIiGjMqEVF0ksk7VHW/0jS+yXt1fnUIiKi19S5UrkeeJGkKcA1wMnApZ1MKiIielOdoiLbvwWOBf7J9geo3gkfERGxlVpFRdLbgQ8DPyixTk6ZHxERPapOUTkTOBv4Tnm3yauoXrQVERGxlVGvOGz/GPixpJeU7fuBT3Y6sYiI6D11Rn+9vbytcX3ZfpOkr3U8s4iI6Dl1bn/9d+Ao4NcAtu8A3tnJpCIiojfVevjR9oPDQls6kEtERPS4OqO4HpT0DsCS9qbqT1nf2bQiIqIX1blSORU4DZgCDAIzy3ZERMRWRi0qth+x/WHbB9o+wPZf2v71aMdJmibpWknrJa2TdEaJ/42kX0laW5ZjWo45W9KApHslHdUSn11iA5LOaokfLOkWSfdJurxcSUVERJfUGf21WNLElu1JkhbVOPezwKdtvw6YBZwmaehJ/AttzyzLinLeQ4ATgNcDs4GvSZogaQLwVeBoqif5T2w5zxfKuWYAjwKn1MgrIiI6pM7trz+x/djQhu1HgTePdpDtDbZvL+tPUvXDTBnhkDnAUttP2/4FMAAcWpYB2/fbfgZYCsyRJOAI4Ipy/GJgbo3fExERHVKnqOwhadLQhqT92MFpWiRNpypEt5TQ6ZLulLSo5dxTgNZRZoMltr34K4HHbD87LB4REV1Sp6h8CfiJpPMknQf8BPiHul8g6aXAt4EzbT8BXAy8mqrDf0M5P4DaHO4xxNvlMF/SGklrNm3aVDf1iIjYQXU66pcAxwEPAxuBY21/o87Jy3tXvg180/aV5XwP295i+zngX6hub0F1pTGt5fCpwEMjxB8BJkrac1i83W9YaLvfdn9fX1+d1CMiYgzqvvnxZ8CVwFXAU5IOGu2A0udxCbDe9pdb4pNbmn0AuLusLwdOkLSPpIOBGcCtwGpgRhnptTdVZ/5y26aa2PK4cvy8kl9ERHTJqH0jkj4BLKC6UtlCddvJwJ+McujhwEeAuyStLbHPUY3emlnO8QDwMYAyA/Iy4B6qkWOn2d5ScjgdWAlMABbZXlfO91lgqaTzgZ9SFbGIiOiSOh3uZwB/XOfZlFa2b6B9v8eKEY65ALigTXxFu+PKjMmHDo9HRER31Ln99SDweKcTiYiI3lfnSuV+4DpJPwCeHgq29pNERERAvaLyy7LsXZaIiIi26rz58fMAkl5i+zedTykiInpV3vwYERGNyZsfIyKiMXnzY0RENCZvfoyIiMbkzY8REdGYEa9UyguyPmL7wzspn4iI6GEjXqmUubfm7KRcIiKix9XpU7lR0j8DlwO/f05l6K2OERERQ+oUlXeUz3NbYqZ6lW9ERMTvjdansgdwse1lOymfiIjoYaP1qTwHnL6TcomIiB5XZ0jxKkn/RdI0SfsNLR3PLCIiek6dPpW/Kp+tz6YYeFXz6URERC+rM0vxwTsjkYiI6H113lF/Uru47SXNpxMREb2szu2vt7Wsvwh4D3A7kKISERFbqXP76xOt25JeAXyjYxlFRETPqjX1/TC/BWaM1qiMFrtW0npJ6ySdUeL7SVol6b7yOanEJekiSQOS7pT0lpZzzSvt75M0ryX+Vkl3lWMukqQx/J6IiGhInTc/fk/S8rJ8H7gXuKrGuZ8FPm37dcAs4DRJhwBnAdfYngFcU7YBjqYqVjOA+cDF5fv3AxYAhwGHAguGClFpM7/luNk18oqIiA6p06fyjy3rzwL/YXtwtINsbwA2lPUnJa2nmj5/DvCu0mwxcB3w2RJfYtvAzZImSppc2q6yvRlA0ipgtqTrgJfbvqnElwBzgatr/KaIiOiAOkXll8AG278DkLSvpOm2H6j7JZKmA28GbgEOLAUH2xskHVCaTQFa3zA5WGIjxQfbxCMiokvq9Kl8C3iuZXtLidUi6aXAt4EzbT8xUtM2MY8h3i6H+ZLWSFqzadOm0VKOiIgxqlNU9rT9zNBGWd+7zskl7UVVUL5p+8oSfrjc1qJ8bizxQWBay+FTgYdGiU9tE9+G7YW2+2339/X11Uk9IiLGoE5R2STp/UMbkuYAj4x2UBmJdQmw3vaXW3YtB4ZGcM3j+U7/5cBJZRTYLODxcptsJXCkpEmlg/5IYGXZ96SkWeW7TqLeAIKIiOiQOn0qpwLfLC/qguoKoe1T9sMcDnwEuEvS2hL7HPD3wDJJp1D11xxf9q0AjgEGqIYtnwxge7Ok84DVpd25Q532wMeBS4F9qTro00kfEdFFdR5+/Dkwq/SNyPaTdU5s+wba93tA9VT+8PZm60krW/ctAha1ia8B3lAnn4iI6Lw6z6n8raSJtp8qQ4MnSTp/ZyQXERG9pU6fytG2HxvasP0o1W2qiIiIrdQpKhMk7TO0IWlfYJ8R2kdExDhVp6P+fwHXSPo61XMgf0X1JHxERMRW6nTU/4OkO4H3ltB5tld2Nq2IiOhFda5UAH4K7EV1pfLTzqUTERG9rM7orw8CtwLHAR8EbpF0XKcTi4iI3lPnSuW/Am+zvRFAUh/wb8AVnUwsIiJ6T53RX3sMFZTi1zWPi4iIcabOlcoPJa0ELivbH6KaUiUiImIrdUZ/fUbSscCfUk27stD2dzqeWURE9Jxao7/KtPVXjtowIiLGtfSNREREY1JUIiKiMdstKpKuKZ9f2HnpRERELxupT2WypD8D3i9pKcPejWL79o5mFhERPWekonIOcBbVu9+/PGyfgSM6lVRERPSm7RYV21cAV0j6b7bP24k5RUREj6rznMp5kt4PvLOErrP9/c6mFRERvajOhJJ/B5wB3FOWM0osIiJiK3UefnwfMNP2cwCSFlNNf392JxOLiIjeU/c5lYkt66/oRCIREdH76hSVvwN+KunScpVyG/C3ox0kaZGkjZLubon9jaRfSVpblmNa9p0taUDSvZKOaonPLrEBSWe1xA+WdIuk+yRdLmnvuj86IiI6Y9SiYvsyYBbV3F9XAm+3vbTGuS8FZreJX2h7ZllWAEg6BDgBeH055muSJkiaAHwVOBo4BDixtAX4QjnXDOBR4JQaOUVERAfVuv1le4Pt5bavsv1/ax5zPbC5Zh5zgKW2n7b9C2AAOLQsA7bvt/0MsBSYI0lUz8kMvShsMTC35ndFRESHdGPur9Ml3Vluj00qsSnAgy1tBktse/FXAo/ZfnZYvC1J8yWtkbRm06ZNTf2OiIgYZmcXlYuBVwMzgQ3Al0pcbdp6DPG2bC+03W+7v6+vb8cyjoiI2kYsKpL2aO1of6FsP2x7Sxme/C9Ut7egutKY1tJ0KvDQCPFHgImS9hwWj4iILhqxqJT/+N8h6aAmvkzS5JbNDwBDBWs5cIKkfSQdDMwAbgVWAzPKSK+9qTrzl9s2cC1wXDl+HnBVEzlGRMTY1Xn4cTKwTtKtwG+GgrbfP9JBki4D3gXsL2kQWAC8S9JMqltVDwAfK+daJ2kZ1RP7zwKn2d5SznM6sBKYACyyva58xWeBpZLOp3oY85I6PzgiIjqnTlH5/FhObPvENuHt/off9gXABW3iK4AVbeL38/zts4iI2AXUmVDyx5L+EJhh+98kvZjqqiEiImIrdSaU/M9Uz4P8zxKaAny3k0lFRERvqjOk+DTgcOAJANv3AQd0MqmIiOhNdYrK0+VpdgDKMN7tPhMSERHjV52i8mNJnwP2lfTnwLeA73U2rYiI6EV1ispZwCbgLqohwCuAv+5kUhER0ZvqjP56rkx5fwvVba97y8OHERERWxm1qEh6H/A/gJ9Tzbl1sKSP2b6608lFRERvqfPw45eAd9seAJD0auAHQIpKRERspU6fysahglLcD2zsUD4REdHDtnulIunYsrpO0gpgGVWfyvFUEz1GRERsZaTbX3/Rsv4w8GdlfRMwadvmEREx3m23qNg+eWcmEhERva/O6K+DgU8A01vbjzb1fUREjD91Rn99l2rK+u8Bz3U2nYiI6GV1isrvbF/U8UwiIqLn1SkqX5G0APgR8PRQ0PbtHcsqIiJ6Up2i8kbgI8ARPH/7y2U7Ypf0y3Pf2O0UxoWDzrmr2ynELqZOUfkA8KrW6e8jIiLaqfNE/R3AxE4nEhERva/OlcqBwM8krWbrPpUMKY6IiK3UKSoLxnJiSYuA/0Q1d9gbSmw/4HKqZ14eAD5o+1FJAr4CHAP8Fvjo0EAASfN4/v0t59teXOJvBS4F9qV6x8sZmZI/IqK7Rr39ZfvH7ZYa574UmD0sdhZwje0ZwDVlG+BoYEZZ5gMXw++L0ALgMOBQYIGkoSliLi5th44b/l0REbGTjVpUJD0p6Ymy/E7SFklPjHac7euBzcPCc4DFZX0xMLclvsSVm4GJkiYDRwGrbG+2/SiwCphd9r3c9k3l6mRJy7kiIqJL6rz58WWt25LmUl01jMWBtjeU826QdECJTwEebGk3WGIjxQfbxNuSNJ/qqoaDDjpojKlHRMRo6oz+2ort79L8Mypq91VjiLdle6Htftv9fX19Y0wxIiJGU2dCyWNbNvcA+hnhP+CjeFjS5HKVMpnnX/Y1CExraTcVeKjE3zUsfl2JT23TPiIiuqjOlcpftCxHAU9S9YGMxXJgXlmfB1zVEj9JlVnA4+U22UrgSEmTSgf9kcDKsu9JSbPKyLGTWs4VERFdUqdPZUzvVZF0GdVVxv6SBqlGcf09sEzSKcAvqd4iCdWQ4GOAAaohxSeX794s6Tyef9PkubaHOv8/zvNDiq8uS0REdNFIrxM+Z4TjbPu8kU5s+8Tt7HpPu5MBp23nPIuARW3ia4A3jJRDRETsXCNdqfymTewlwCnAK4ERi0pERIw/I71O+EtD65JeBpxBdVtqKfCl7R0XERHj14h9KuWJ9k8BH6Z6WPEt5SHEiIiIbYzUp/JF4FhgIfBG20/ttKwiIqInjTSk+NPAH1BN5vhQy1QtT9aZpiUiIsafkfpUdvhp+4iIGN9SOCIiojEpKhER0ZgUlYiIaEyKSkRENCZFJSIiGpOiEhERjUlRiYiIxqSoREREY1JUIiKiMSkqERHRmBSViIhoTIpKREQ0JkUlIiIak6ISERGNSVGJiIjGdKWoSHpA0l2S1kpaU2L7SVol6b7yOanEJekiSQOS7pT0lpbzzCvt75M0rxu/JSIintfNK5V3255pu79snwVcY3sGcE3ZBjgamFGW+cDFUBUhYAFwGHAosGCoEEVERHfsSre/5gCLy/piYG5LfIkrNwMTJU0GjgJW2d5s+1FgFTB7ZycdERHP61ZRMfAjSbdJml9iB9reAFA+DyjxKcCDLccOltj24hER0SXbfUd9hx1u+yFJBwCrJP1shLZqE/MI8W1PUBWu+QAHHXTQjuYaERE1deVKxfZD5XMj8B2qPpGHy20tyufG0nwQmNZy+FTgoRHi7b5voe1+2/19fX1N/pSIiGix04uKpJdIetnQOnAkcDewHBgawTUPuKqsLwdOKqPAZgGPl9tjK4EjJU0qHfRHllhERHRJN25/HQh8R9LQ9/+r7R9KWg0sk3QK8Evg+NJ+BXAMMAD8FjgZwPZmSecBq0u7c21v3nk/IyIihtvpRcX2/cCb2sR/DbynTdzAads51yJgUdM5RkTE2OxKQ4ojIqLHpahERERjujWkuCe89TNLup3Cbu+2L57U7RQiokG5UomIiMakqERERGNSVCIiojEpKhER0ZgUlYiIaEyKSkRENCZFJSIiGpOiEhERjUlRiYiIxqSoREREY1JUIiKiMSkqERHRmBSViIhoTIpKREQ0JkUlIiIak6ISERGNSVGJiIjGpKhERERjer6oSJot6V5JA5LO6nY+ERHjWU8XFUkTgK8CRwOHACdKOqS7WUVEjF89XVSAQ4EB2/fbfgZYCszpck4REeNWrxeVKcCDLduDJRYREV2wZ7cTeIHUJuZtGknzgfll8ylJ93Y0q+7aH3ik20nUpX+c1+0UdiU99bcDYEG7fwXHrZ76++mTO/y3+8M6jXq9qAwC01q2pwIPDW9keyGwcGcl1U2S1tju73YesePyt+tt+ftVev3212pghqSDJe0NnAAs73JOERHjVk9fqdh+VtLpwEpgArDI9roupxURMW71dFEBsL0CWNHtPHYh4+I2324qf7velr8fIHubfu2IiIgx6fU+lYiI2IWkqOwmMl1N75K0SNJGSXd3O5fYMZKmSbpW0npJ6ySd0e2cui23v3YDZbqa/wP8OdUw69XAibbv6WpiUYukdwJPAUtsv6Hb+UR9kiYDk23fLullwG3A3PH8716uVHYPma6mh9m+Htjc7Txix9neYPv2sv4ksJ5xPqtHisruIdPVRHSZpOnAm4FbuptJd6Wo7B5qTVcTEZ0h6aXAt4EzbT/R7Xy6KUVl91BrupqIaJ6kvagKyjdtX9ntfLotRWX3kOlqIrpAkoBLgPW2v9ztfHYFKSq7AdvPAkPT1awHlmW6mt4h6TLgJuCPJQ1KOqXbOUVthwMfAY6QtLYsx3Q7qW7KkOKIiGhMrlQiIqIxKSoREdGYFJWIiGhMikpERDQmRSUiIhqTohIREY1JUYkAJP2kRpszJb24w3nMHO05B0kflfTPDX9v4+eM8SlFJQKw/Y4azc4EdqiolNcS7IiZwLh+eC56W4pKBCDpqfL5LknXSbpC0s8kfVOVTwJ/AFwr6drS9khJN0m6XdK3yqSCSHpA0jmSbgCOl/RqST+UdJuk/y3ptaXd8ZLulnSHpOvLFDvnAh8qT2Z/qEbefZK+LWl1WQ6XtEfJYWJLuwFJB7Zr3/g/zBjX9ux2AhG7oDcDr6ealPNG4HDbF0n6FPBu249I2h/4a+C9tn8j6bPAp6iKAsDvbP8pgKRrgFNt3yfpMOBrwBHAOcBRtn8laaLtZySdA/TbPr1mrl8BLrR9g6SDgJW2XyfpKuADwNfLdz5g+2FJ/zq8PfC6F/jPK+L3UlQitnWr7UEASWuB6cANw9rMAg4BbqzmFGRvqvm7hlxejn8p8A7gW6UdwD7l80bgUknLgLHObvte4JCWc7+8vIHwcqqi9XWqCUYvH6V9RCNSVCK29XTL+hba/3siYJXtE7dzjt+Uzz2Ax2zPHN7A9qnlKuJ9wFpJ27SpYQ/g7bb/31bJSTcBr5HUB8wFzh+l/Ri+OmJb6VOJqO9JYOj/6m8GDpf0GgBJL5b0R8MPKC9s+oWk40s7SXpTWX+17VtsnwM8QvVOnNbvqONHVDNUU845s3yvge8AX6aalv3XI7WPaEqKSkR9C4GrJV1rexPwUeAySXdSFZnXbue4DwOnSLoDWAfMKfEvSrpL0t3A9cAdwLVUt6dqddQDnwT6Jd0p6R7g1JZ9lwN/yfO3vkZrH/GCZer7iIhoTK5UIiKiMemoj9hFSToZOGNY+Ebbp3Ujn4g6cvsrIiIak9tfERHRmBSViIhoTIpKREQ0JkUlIiIak6ISERGN+f+UQGoL+CmGNwAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0xad29d30>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "#interest_level 分布，看看各类样本分布是否均衡\n",
    "sns.countplot(train.interest_level);\n",
    "pyplot.xlabel('interest_level');\n",
    "pyplot.ylabel('Number of occurrences');"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "样本分布不均匀，所以交叉验证时也考虑各类样本按比例抽取，特征工程文中已经做好不再做。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 158,
   "metadata": {},
   "outputs": [],
   "source": [
    "# 划分X和y\n",
    "y_train = train[\"interest_level\"]\n",
    "X_train = train.drop([\"interest_level\"], axis=1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "# 交叉验证\n",
    "kfold = StratifiedKFold(n_splits=5, shuffle=True, random_state=3)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 第一轮参数调整得到的n_estimators最优值（340），其余参数继续默认值"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "用交叉验证评价模型性能时，用scoring参数定义评价指标。评价指标是越高越好，因此用一些损失函数当评价指标时，需要再加负号，如neg_log_loss，neg_mean_squared_error 详见sklearn文档：http://scikit-learn.org/stable/modules/model_evaluation.html#log-loss"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'max_depth': range(3, 10, 2), 'min_child_weight': range(1, 6, 2)}"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#max_depth 建议3-10， min_child_weight=1／sqrt(ratio_rare_event) =5.5\n",
    "max_depth = range(3,10,2)\n",
    "min_child_weight = range(1,6,2)\n",
    "param_test2_1 = dict(max_depth=max_depth, min_child_weight=min_child_weight)\n",
    "param_test2_1"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 50,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "D:\\anaconda\\lib\\site-packages\\sklearn\\model_selection\\_search.py:761: DeprecationWarning: The grid_scores_ attribute was deprecated in version 0.18 in favor of the more elaborate cv_results_ attribute. The grid_scores_ attribute will not be available from 0.20\n",
      "  DeprecationWarning)\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "([mean: -0.59352, std: 0.00344, params: {'max_depth': 3, 'min_child_weight': 1},\n",
       "  mean: -0.59364, std: 0.00321, params: {'max_depth': 3, 'min_child_weight': 3},\n",
       "  mean: -0.59354, std: 0.00301, params: {'max_depth': 3, 'min_child_weight': 5},\n",
       "  mean: -0.58215, std: 0.00367, params: {'max_depth': 5, 'min_child_weight': 1},\n",
       "  mean: -0.58216, std: 0.00351, params: {'max_depth': 5, 'min_child_weight': 3},\n",
       "  mean: -0.58241, std: 0.00410, params: {'max_depth': 5, 'min_child_weight': 5},\n",
       "  mean: -0.58605, std: 0.00412, params: {'max_depth': 7, 'min_child_weight': 1},\n",
       "  mean: -0.58463, std: 0.00376, params: {'max_depth': 7, 'min_child_weight': 3},\n",
       "  mean: -0.58389, std: 0.00421, params: {'max_depth': 7, 'min_child_weight': 5},\n",
       "  mean: -0.61061, std: 0.00440, params: {'max_depth': 9, 'min_child_weight': 1},\n",
       "  mean: -0.60097, std: 0.00477, params: {'max_depth': 9, 'min_child_weight': 3},\n",
       "  mean: -0.59848, std: 0.00396, params: {'max_depth': 9, 'min_child_weight': 5}],\n",
       " {'max_depth': 5, 'min_child_weight': 1},\n",
       " -0.5821542448469158)"
      ]
     },
     "execution_count": 50,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#使用第一轮参数得到的n_estimators=340进行训练\n",
    "xgb2_1 = XGBClassifier(\n",
    "        learning_rate =0.1,\n",
    "        n_estimators=340,  #第一轮参数调整得到的n_estimators最优值\n",
    "        max_depth=5,\n",
    "        min_child_weight=1,\n",
    "        gamma=0,\n",
    "        subsample=0.7,\n",
    "        colsample_bytree=0.8,\n",
    "        colsample_bylevel = 0.7,\n",
    "        objective= 'multi:softprob',\n",
    "        seed=3)\n",
    "\n",
    "\n",
    "gsearch2_1 = GridSearchCV(xgb2_1, param_grid = param_test2_1, scoring='neg_log_loss',n_jobs=-1, cv=kfold)\n",
    "gsearch2_1.fit(X_train , y_train)\n",
    "\n",
    "gsearch2_1.grid_scores_, gsearch2_1.best_params_,     gsearch2_1.best_score_"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 61,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "D:\\anaconda\\lib\\site-packages\\sklearn\\utils\\deprecation.py:122: FutureWarning: You are accessing a training score ('mean_train_score'), which will not be available by default any more in 0.21. If you need training scores, please set return_train_score=True\n",
      "  warnings.warn(*warn_args, **warn_kwargs)\n",
      "D:\\anaconda\\lib\\site-packages\\sklearn\\utils\\deprecation.py:122: FutureWarning: You are accessing a training score ('split0_train_score'), which will not be available by default any more in 0.21. If you need training scores, please set return_train_score=True\n",
      "  warnings.warn(*warn_args, **warn_kwargs)\n",
      "D:\\anaconda\\lib\\site-packages\\sklearn\\utils\\deprecation.py:122: FutureWarning: You are accessing a training score ('split1_train_score'), which will not be available by default any more in 0.21. If you need training scores, please set return_train_score=True\n",
      "  warnings.warn(*warn_args, **warn_kwargs)\n",
      "D:\\anaconda\\lib\\site-packages\\sklearn\\utils\\deprecation.py:122: FutureWarning: You are accessing a training score ('split2_train_score'), which will not be available by default any more in 0.21. If you need training scores, please set return_train_score=True\n",
      "  warnings.warn(*warn_args, **warn_kwargs)\n",
      "D:\\anaconda\\lib\\site-packages\\sklearn\\utils\\deprecation.py:122: FutureWarning: You are accessing a training score ('split3_train_score'), which will not be available by default any more in 0.21. If you need training scores, please set return_train_score=True\n",
      "  warnings.warn(*warn_args, **warn_kwargs)\n",
      "D:\\anaconda\\lib\\site-packages\\sklearn\\utils\\deprecation.py:122: FutureWarning: You are accessing a training score ('split4_train_score'), which will not be available by default any more in 0.21. If you need training scores, please set return_train_score=True\n",
      "  warnings.warn(*warn_args, **warn_kwargs)\n",
      "D:\\anaconda\\lib\\site-packages\\sklearn\\utils\\deprecation.py:122: FutureWarning: You are accessing a training score ('std_train_score'), which will not be available by default any more in 0.21. If you need training scores, please set return_train_score=True\n",
      "  warnings.warn(*warn_args, **warn_kwargs)\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "{'mean_fit_time': array([ 307.22790408,  311.24719081,  312.99544158,  452.32521734,\n",
       "         452.73992071,  451.47395887,  594.57516208,  595.28805985,\n",
       "         592.24236445,  744.62945495,  736.39316278,  567.35393009]),\n",
       " 'mean_score_time': array([ 0.58415322,  0.58435378,  0.5895678 ,  1.05039268,  1.24330478,\n",
       "         1.0574111 ,  2.9137464 ,  3.7359302 ,  3.27490706,  4.30364032,\n",
       "         5.07328687,  2.79483137]),\n",
       " 'mean_test_score': array([-0.59352338, -0.59363606, -0.5935351 , -0.58215424, -0.58216092,\n",
       "        -0.58240947, -0.58605197, -0.58463279, -0.58389269, -0.61060801,\n",
       "        -0.60096525, -0.59847864]),\n",
       " 'mean_train_score': array([-0.56172457, -0.56290576, -0.56336552, -0.47181169, -0.47927137,\n",
       "        -0.48278717, -0.32786119, -0.35327904, -0.36697939, -0.17219707,\n",
       "        -0.21517912, -0.2416077 ]),\n",
       " 'param_max_depth': masked_array(data = [3 3 3 5 5 5 7 7 7 9 9 9],\n",
       "              mask = [False False False False False False False False False False False False],\n",
       "        fill_value = ?),\n",
       " 'param_min_child_weight': masked_array(data = [1 3 5 1 3 5 1 3 5 1 3 5],\n",
       "              mask = [False False False False False False False False False False False False],\n",
       "        fill_value = ?),\n",
       " 'params': [{'max_depth': 3, 'min_child_weight': 1},\n",
       "  {'max_depth': 3, 'min_child_weight': 3},\n",
       "  {'max_depth': 3, 'min_child_weight': 5},\n",
       "  {'max_depth': 5, 'min_child_weight': 1},\n",
       "  {'max_depth': 5, 'min_child_weight': 3},\n",
       "  {'max_depth': 5, 'min_child_weight': 5},\n",
       "  {'max_depth': 7, 'min_child_weight': 1},\n",
       "  {'max_depth': 7, 'min_child_weight': 3},\n",
       "  {'max_depth': 7, 'min_child_weight': 5},\n",
       "  {'max_depth': 9, 'min_child_weight': 1},\n",
       "  {'max_depth': 9, 'min_child_weight': 3},\n",
       "  {'max_depth': 9, 'min_child_weight': 5}],\n",
       " 'rank_test_score': array([ 7,  9,  8,  1,  2,  3,  6,  5,  4, 12, 11, 10]),\n",
       " 'split0_test_score': array([-0.58798653, -0.58817796, -0.58872024, -0.57651759, -0.57680177,\n",
       "        -0.57573856, -0.57915874, -0.57767245, -0.57630328, -0.60287767,\n",
       "        -0.59518218, -0.59352958]),\n",
       " 'split0_train_score': array([-0.56307002, -0.56438846, -0.5650962 , -0.47298586, -0.4808219 ,\n",
       "        -0.48548642, -0.32907145, -0.35513135, -0.37148668, -0.17365029,\n",
       "        -0.21712119, -0.24244012]),\n",
       " 'split1_test_score': array([-0.59199525, -0.59244943, -0.5922467 , -0.58001192, -0.57993877,\n",
       "        -0.58053717, -0.58465319, -0.5845056 , -0.58350624, -0.61064176,\n",
       "        -0.59782962, -0.59576847]),\n",
       " 'split1_train_score': array([-0.56172755, -0.5632706 , -0.56360579, -0.47186701, -0.47874685,\n",
       "        -0.48248145, -0.32674353, -0.35298371, -0.3663181 , -0.17167868,\n",
       "        -0.21363106, -0.24077197]),\n",
       " 'split2_test_score': array([-0.59336152, -0.59393675, -0.5932765 , -0.58210032, -0.5825464 ,\n",
       "        -0.58334031, -0.5880511 , -0.58627189, -0.58434187, -0.61051658,\n",
       "        -0.59991042, -0.59702751]),\n",
       " 'split2_train_score': array([-0.56132676, -0.56278419, -0.56358316, -0.47226326, -0.47937135,\n",
       "        -0.48262107, -0.32610357, -0.35188204, -0.36546652, -0.17284635,\n",
       "        -0.21583119, -0.24222837]),\n",
       " 'split3_test_score': array([-0.59744694, -0.59702027, -0.59703019, -0.58568828, -0.58477569,\n",
       "        -0.58443128, -0.5867912 , -0.58581047, -0.5865665 , -0.61269664,\n",
       "        -0.60282372, -0.60179516]),\n",
       " 'split3_train_score': array([-0.56147266, -0.56203818, -0.56259398, -0.47161722, -0.47904284,\n",
       "        -0.48176874, -0.32853647, -0.35183658, -0.36468048, -0.17188446,\n",
       "        -0.21449417, -0.24047296]),\n",
       " 'split4_test_score': array([-0.59682769, -0.59659682, -0.59640276, -0.58645442, -0.58674334,\n",
       "        -0.58800171, -0.59160729, -0.58890486, -0.58874705, -0.61630914,\n",
       "        -0.60908276, -0.60427422]),\n",
       " 'split4_train_score': array([-0.56102586, -0.56204738, -0.56194846, -0.47032508, -0.47837392,\n",
       "        -0.48157818, -0.32885094, -0.3545615 , -0.36694515, -0.17092557,\n",
       "        -0.21481797, -0.24212506]),\n",
       " 'std_fit_time': array([  2.26198991,   1.20788766,   0.69125426,   0.86177751,\n",
       "          1.34093876,   0.59503935,   1.46693775,   2.78529835,\n",
       "          0.99849074,   0.33434459,   9.74357259,  52.65316056]),\n",
       " 'std_score_time': array([ 0.01413176,  0.01453572,  0.0209364 ,  0.21060143,  0.37941415,\n",
       "         0.0980905 ,  1.74786366,  1.8725814 ,  1.04395838,  1.59684315,\n",
       "         2.03023943,  0.71935392]),\n",
       " 'std_test_score': array([ 0.00344474,  0.00320963,  0.00301161,  0.00366998,  0.00351113,\n",
       "         0.00410438,  0.00412004,  0.00376259,  0.004211  ,  0.00439611,\n",
       "         0.00476914,  0.00396299]),\n",
       " 'std_train_score': array([ 0.00070995,  0.00087588,  0.00106805,  0.00087561,  0.00084225,\n",
       "         0.00140754,  0.00120322,  0.00135612,  0.00238025,  0.00095043,\n",
       "         0.00119941,  0.00081632])}"
      ]
     },
     "execution_count": 61,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "gsearch2_1.cv_results_"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 62,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Best: -0.582154 using {'max_depth': 5, 'min_child_weight': 1}\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "D:\\anaconda\\lib\\site-packages\\sklearn\\utils\\deprecation.py:122: FutureWarning: You are accessing a training score ('mean_train_score'), which will not be available by default any more in 0.21. If you need training scores, please set return_train_score=True\n",
      "  warnings.warn(*warn_args, **warn_kwargs)\n",
      "D:\\anaconda\\lib\\site-packages\\sklearn\\utils\\deprecation.py:122: FutureWarning: You are accessing a training score ('std_train_score'), which will not be available by default any more in 0.21. If you need training scores, please set return_train_score=True\n",
      "  warnings.warn(*warn_args, **warn_kwargs)\n",
      "D:\\anaconda\\lib\\site-packages\\sklearn\\utils\\deprecation.py:122: FutureWarning: You are accessing a training score ('split0_train_score'), which will not be available by default any more in 0.21. If you need training scores, please set return_train_score=True\n",
      "  warnings.warn(*warn_args, **warn_kwargs)\n",
      "D:\\anaconda\\lib\\site-packages\\sklearn\\utils\\deprecation.py:122: FutureWarning: You are accessing a training score ('split1_train_score'), which will not be available by default any more in 0.21. If you need training scores, please set return_train_score=True\n",
      "  warnings.warn(*warn_args, **warn_kwargs)\n",
      "D:\\anaconda\\lib\\site-packages\\sklearn\\utils\\deprecation.py:122: FutureWarning: You are accessing a training score ('split2_train_score'), which will not be available by default any more in 0.21. If you need training scores, please set return_train_score=True\n",
      "  warnings.warn(*warn_args, **warn_kwargs)\n",
      "D:\\anaconda\\lib\\site-packages\\sklearn\\utils\\deprecation.py:122: FutureWarning: You are accessing a training score ('split3_train_score'), which will not be available by default any more in 0.21. If you need training scores, please set return_train_score=True\n",
      "  warnings.warn(*warn_args, **warn_kwargs)\n",
      "D:\\anaconda\\lib\\site-packages\\sklearn\\utils\\deprecation.py:122: FutureWarning: You are accessing a training score ('split4_train_score'), which will not be available by default any more in 0.21. If you need training scores, please set return_train_score=True\n",
      "  warnings.warn(*warn_args, **warn_kwargs)\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAZIAAAELCAYAAADz6wBxAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMS4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvNQv5yAAAIABJREFUeJzt3Xl8VNX5+PHPM5PJRkggLCZsggIVhYgSEbVYUEFQiyJUQUGwLq1LrfUnRVtbl+JKRbSltmpVQBFEiuKGgOLyxSqCAiJUiLiFsIctLFmf3x/3Jkwmk2SSyUwS8rxfr3ll5txz7n3uTTLP3HPPnCuqijHGGFNbnvoOwBhjTONmicQYY0xYLJEYY4wJiyUSY4wxYbFEYowxJiyWSIwxxoTFEokxxpiwWCIxxhgTFkskxhhjwhJT3wFEQ+vWrbVz5871HYYxxjQqK1eu3Kmqbaqr1yQSSefOnVmxYkV9h2GMMY2KiHwfSj3r2jLGGBMWSyTGGGPCYonEGGNMWJrENRJjTOgKCwvJzs7m8OHD9R2KiZL4+Hg6dOiAz+erVXtLJMaYcrKzs2nevDmdO3dGROo7HBNhqsquXbvIzs6mS5cutVqHdW0ZY8o5fPgwrVq1siTSRIgIrVq1CusM1BKJMaYCSyJNS7i/b0skVdj//vvkffghWlJS36EYY0yDZddIqpD77HMcXL6c2OOOI/WqsaRcfDGehIT6DssYYxqUiJ6RiMgQEflaRLJE5I5K6lwmIutE5CsRmeVXvlBE9ojIGwH1u4jIpyKyUUTmiEhspOLv9MzTtHvkYTzx8Wy9516yBgxk+5THKNy2LVKbNKbJ27NnD//4xz9q1Xbq1KkcPHiwjiOqGwMGDKj1DBuvvvoq69atq/G6fvnLX9K2bVt69uxZq+2GKmKJRES8wDRgKHAiMFpETgyo0w24EzhLVU8CbvVbPBkYG2TVDwOPqWo3YDdwTQTCd+KLjSVl2DA6z3uFY1+YSWLf09j19NNknXsem2+fwKEv10Zq08Y0WUdrIglHYCIJ1fjx41m4cGEEIiovkl1bfYEsVd0EICKzgYsB/6NxHTBNVXcDqOr20gWq+q6IDPBfoThXhM4BrnCLpgP3AE9GZhfKtktiZiaJmZkU/Pgju194gT2vzGPfG2+QcOqppI4bR/PzzkW83kiGYUzU3fv6V6zL2Ven6zyxXTJ3//ykSpffcccdfPPNN/Tu3ZtBgwbRtm1bXn75ZfLz8xk+fDj33nsvBw4c4LLLLiM7O5vi4mL+9Kc/sW3bNnJychg4cCCtW7dm6dKlQdeflJTETTfdxJIlS2jZsiUPPPAAv//97/nhhx+YOnUqw4YN47vvvmPs2LEcOHAAgL///e+ceeaZzJ8/n2nTprF48WK2bt3Kz372Mz788EPS0tIqbOfQoUNcffXVrFu3jh49enDo0KGyZYsWLeLuu+8mPz+f448/nueee46kpCQ6d+7M5ZdfXhb7rFmz2L59OwsWLOCDDz5g0qRJzJs3D4C5c+dy4403smfPHv7973/Tv3//CjGcffbZfPfddyH/bmorkl1b7YEf/V5nu2X+ugPdRWSZiHwiIkOqWWcrYI+qFlWxzoiK7diRY+68k64fvE/bOyZStG0bm3/7W74ZfD67nn+e4v37oxmOMUedhx56iOOPP55Vq1YxaNAgNm7cyPLly1m1ahUrV67kww8/ZOHChbRr147Vq1ezdu1ahgwZwi233EK7du1YunRppUkE4MCBAwwYMICVK1fSvHlz7rrrLhYvXsz8+fP585//DEDbtm1ZvHgxn3/+OXPmzOGWW24BYPjw4aSlpTFt2jSuu+467r333qBJBODJJ58kMTGRNWvW8Mc//pGVK1cCsHPnTiZNmsSSJUv4/PPPyczMZMqUKWXtkpOTWb58OTfffDO33norZ555JsOGDWPy5MmsWrWK448/HoCioiKWL1/O1KlTuffeewHIycnhggsuCP+XUEORPCMJNp5Mg2y/GzAA6AB8JCI9VXVPGOt0KopcD1wP0KlTp1DirRFvUhKtxo8ndexY9r/7LrnTZ7D9oYfZ+be/kzLiUlLHjiW2Y8c6364x0VTVmUM0LFq0iEWLFnHKKacAkJeXx8aNG+nfvz+33347EydO5KKLLgr6abwysbGxDBnifGbt1asXcXFx+Hw+evXqVfbpvbCwkJtvvplVq1bh9XrZsGFDWfu//e1v9OzZk379+jF69OhKt/Phhx+WJaCMjAwyMjIA+OSTT1i3bh1nnXUWAAUFBZxxxhll7UrXOXr0aH73u99Vuv5LL70UgD59+pTF3a5dO956662Qj0VdiWQiyQb830k7ADlB6nyiqoXAtyLyNU5i+aySde4EWohIjHtWEmydAKjqU8BTAJmZmUGTTV0Qr5fkwYNJHjyYQ1+uJXfGDHbPeondM18g6dxzaDVuHAmZmTYu35haUFXuvPNOfvWrX1VYtnLlSt566y3uvPNOBg8eXHY2UR2fz1f2/+jxeIiLiyt7XlTkdHY89thjHHPMMaxevZqSkhLi4+PL2m/evBmPx8O2bdsoKSnB46m8YyfY/72qMmjQIF566aVq21T1vlEat9frLYu7vkSya+szoJs7yioWGAUsCKjzKjAQQERa43R1bapshaqqwFJgpFs0DnitjuOutYRePWk/+RG6vruEVtddx6HPVvD92Kv4bsRI9r72GlpQUN8hGtPgNW/enP1uF/H555/Ps88+S15eHuC8iW/fvp2cnBwSExMZM2YMt99+O59//nmFtuHYu3cv6enpeDweZs6cSXFxMeB0J1199dXMmjWLHj16lOuSCnT22Wfz4osvArB27VrWrFkDQL9+/Vi2bBlZWVkAHDx4sNwZz5w5c8p+lp6p1NV+RUrEEol7xnAz8A6wHnhZVb8SkftEZJhb7R1gl4isw0kQE1R1F4CIfATMBc4VkWwROd9tMxG4TUSycK6Z/DtS+1BbvmOOoe1tv6Pr+0tJu+ceSg4fJmfiHWSdex47//lPinbvru8QjWmwWrVqxVlnnUXPnj1ZvHgxV1xxBWeccQa9evVi5MiR7N+/ny+//JK+ffvSu3dv7r//fu666y4Arr/+eoYOHcrAgQPDiuHGG29k+vTp9OvXjw0bNtCsWTMAHnjgAfr370///v2ZMmUKzzzzDOvXrw+6jhtuuIG8vDwyMjJ45JFH6Nu3LwBt2rTh+eefZ/To0WRkZNCvXz/+97//lbXLz8/n9NNP5/HHH+exxx4DYNSoUUyePJlTTjmFb775ptK4A6+RjB49mjPOOIOvv/6aDh068O9/R+btUpwP+Ue3zMxMrc87JGpJCQeWLSP3+ekcWLYMiYsjZdgwUsddRVzXrvUWlzHBrF+/nh49etR3GE1S6d1cW7duHfVtB/u9i8hKVc2srq19sz0KxOMhqX9/kvr3J3/jRnJnzGTvggXsmTuXZj/9KanjrqLZT39q11GMMY2SJZIoi+vWjfS/3Eeb393KnjlzyJ01ix+vu57Y448n9aqrSLl4GB6/C3vGmNo5/fTTyc/PL1c2c+ZMevXqVafbeeedd5g4cWK5si5dujB//vwarysa3/mIBOvaqmdaUMC+t99m1/Tp5K9bj7dFC1pcfjktr7gC3zFt6zs80wRZ11bTFE7Xls3+W88kNpaUiy+my7x5HDtzBgmZfdj11FNknXsumyf8nkNrv6rvEI0xpkrWtdVAiAiJp51G4mmnUfDDD+TOfIG98+ax7/XXScjsQ+pVV9H8XJuGxRjT8NgZSQMU26kTaX/8gzMNy8SJFOVsYfMtv+Wb84eQO306xe6YemOMaQgskTRg3ubNaXX1eI5f9A7tH3+cmLZt2fbgQ2T9bADbHnyQguzs+g7RGGMskTQGEhND8vmD6TzrRTrPfZmkgQPJfXEW3ww+n+zf3MLBFStoCoMmTNNwtE4jH+37kfz4448MHDiQHj16cNJJJ/H444/XatuhsETSyCT06kX7v06m65LFtLrmGg4sX873Y8by3chfsPf1120aFtPoHa2JJBy1uR9JTEwMjz76KOvXr+eTTz5h2rRptbqnSUjbishaTcT50tJo+/9uo/UNv2bvggXkTp9BzoTfs33yX2l5xRW0uPwyYlq2rO8wTWP39h2w9cu6XWdaLxj6UKWL7X4kdXM/kvT0dNLT0wFnrq4ePXqwefNmTjyx3P0F64SdkTRynsREWo4axXFvvkHHp/5FXNeu7Jg6layB57Dl7nvIr2JeHmMaIrsfSd3fj+S7777jiy++4PTTT6/Fb6R6dkZylBCPh6Szzybp7LM5vGEDuTNmsHf+fPbMmUOz/v1JHTeOZmedadOwmJqp4swhGux+JOHfjyQvL48RI0YwdepUkpOTqz0+tWGJ5CgU37077SZNou1tt7F79mx2z3qJH6+9ltiu7jQsw2waFtM42P1IwrsfSWFhISNGjODKK68sSzyRYF1bR7GY1FTa3HgjXd97l/SHHkR8sWz9891kDRjI9qlTKdy+vb5DNKYCux9J3dyPRFW55ppr6NGjB7fddluN2taUJZImwBMbS4tLLqHLf+bRafp0Evr0Yde/niLr3PPImTiRQ1/ZNCym4bD7kdTN/UiWLVvGzJkzee+99+jduze9e/eO2G14bdLGJqrg++/JnfkCe/7zH/TgQRIzM0kdP46kgQNtGpYmziZtrD+N9X4kdkbSRMUeeyxpd/2Rbu8vpe3vf09Bzmayb/4N3wwZSu6MGRTnHajvEI0xjYRdbG/ivMnJtPrl1aReNZb9S94ld/p0tj3wIDue+BstRo6k5ZgxxHZoX99hGlNjdj+S6LGuLVPBoTVryH1+OvveeQdUaX7eeaSOH0fCKafY8OEmwLq2mibr2jJ1KiEjg/ZTHqXru0todc0vOfDpp3x/xZV894vL2Pv6G2hhYX2HaIxpQCyRmEo507D8P7otfY+0u/9MSV4eORMmkHXeIHY+9TTFe/bUd4jGmAbAEomplicxkZajR3PcW2/S4Z9PEnf8ceyYMoWNAway5Z57yN+0qb5DNMbUI0skJmTi8dB8wAA6PfssXV57jeQLL2Dvf+az6YIL+eH668lbtsymszdhO1pn/432NPKHDx+mb9++nHzyyZx00kncfffdtdp2KCyRmFqJ/0l32t1/P12Xvkfr39zM4XXr+fGaa/l22DB2z51LyeHD9R2iaaSO1kQSjtpMIx8XF8d7773H6tWrWbVqFQsXLuSTTz6JSHyWSExYYlq1os1NNznTsDz4IHhj2PqnP5M18Bx2PPEERTt21HeIppHxn0Z+woQJTJ48mdNOO42MjIyyT9UHDhzgwgsv5OSTT6Znz57MmTOHJ554omwa+aq+2Z6UlMTEiRPp06cP5513HsuXL2fAgAEcd9xxLFiwAHCG4fbv359TTz2VU089lY8//hiA+fPnc95556GqbNmyhe7du7N169ag2zl06BCjRo0iIyODyy+/vMI08meccQannnoqv/jFL8qmgOncuTMTJ06kb9++9O3bl6ysLD7++GMWLFjAhAkT6N27d9k32+fOnUvfvn3p3r07H330UYXtiwhJSUmAM+dWYWFhxEZd2vdITJ3wxMbSYvglpFxyMQc/XU7ujBnsfPKf7Hz6GVIuuIDU8eOItyGljc7Dyx/mf7n/q75iDZyQegIT+06sdPlDDz3E2rVrWbVqFYsWLeKVV15h+fLlqCrDhg3jww8/ZMeOHbRr144333wTcObGSklJYcqUKSxdurTKb4aXTiP/8MMPM3z48LJp5NetW8e4ceMYNmxY2TTy8fHxbNy4kdGjR7NixQqGDx/OvHnzmDZtGgsXLgx5Gvk1a9Zw6qmnAuWnkW/WrBkPP/wwU6ZMKZt0snQa+RkzZnDrrbfyxhtvMGzYMC666CJGjhxZtv7SaeTfeust7r33XpYsWUJOTg7XXntt2VQoxcXF9OnTh6ysLG666abGOY28iAwBHge8wDOqWmFOahG5DLgHUGC1ql7hlo8D7nKrTVLV6W75+0A6UJreB6uqzT7YQIgIzfqdTrN+p1Pw3XfONCzz57P3tddIPO00ZxqWAQNsGhYTEptGPrxp5L1eL6tWrWLPnj0MHz6ctWvX0rNnz5COU01ELJGIiBeYBgwCsoHPRGSBqq7zq9MNuBM4S1V3i0hbtzwVuBvIxEkwK922u92mV6qqfcOwgYvt3Jm0P91Fm9/ewp65r5D74gtk33Qzvk6dSB07lpThw/EmNavvME0VqjpziAabRj68aeRLtWjRggEDBrBw4cKIJJJIXiPpC2Sp6iZVLQBmAxcH1LkOmFaaIPzOLM4HFqtqrrtsMTAkgrGaCPImJ9Pqml/SddEi2j82hZjUVLbdfz9ZAwey7eFHKNy8ub5DNA2ITSNfN9PI79ixgz3ud70OHTrEkiVLOOGEE2q0jlBFsmurPfCj3+tsILCDrjuAiCzD6f66R1UXVtLWf8Kn50SkGJiH0+1VYcypiFwPXA/QqVOn8PbE1AmJiSF56FCShw7l0OrV5E6fTu6MGeROn07zQYNIHTeOhFN62zQsTZz/NPJDhw4tm0YenAvlL7zwAllZWUyYMAGPx4PP5+PJJ58Ejkwjn56eXuXtdqtz4403MmLECObOncvAgQODTiPfu3dvTjvtNC688MKgU8rccMMNXH311WRkZNC7d++g08iXzgU2adIkunfvDhyZRr6kpKTsrGXUqFFcd911PPHEE7zyyiuVxu1/jWTLli2MGzeO4uJiSkpKuOyyy7joootqfUyqErG5tkTkF8D5qnqt+3os0FdVf+NX5w2gELgM6AB8BPTEOVOJU9VJbr0/AQdV9VERaa+qm0WkOU4ieUFVZ1QVi8211XAVbtnC7hdfZPfLcynZt4/4jAxSr7qK5PMHIz5ffYfXJNlcW/XHppGvKBvo6Pe6A5ATpM5rqlqoqt8CXwPdqmqrqpvdn/uBWThdaKaR8qWn0/b22+m29D2O+fOfKNm7l5zbbydr0GB2Pm3TsBjTGESya+szoJuIdAE2A6OAKwLqvAqMBp4XkdY4XV2bgG+AB0SkpVtvMHCniMQALVR1p4j4gIuAJRHcBxMlnmbNSL3iClqOGkXeBx+QO2MGOx6dws5/PEnKJReTOvYq4o7rUt9hmkbEppGPnoglElUtEpGbgXdwrn88q6pfich9wApVXeAuGywi64BiYIKq7gIQkb/gJCOA+1Q1V0SaAe+4ScSLk0SejtQ+mOgTj4fmAwfSfOBADn/9NbnTZ7D3lXnseWk2zX52Nq3GjSPxjDPsOoqp1qeffhqV7Zx//vmcf/75UdlWQ2X3IzENXtHOneyePYfdL71E8a5dxHXvTuq4q0i+6CI87hBIU3fsGknT1FCvkRhTJ2Jat6bNze40LPffDyJs+eNd7jQsf6No5876DtGYJs0SiWk0PHFxtBhxKV1enU+n558jISODnf/4B1kDzyHnjjs5/L+6ncrDGBMam2vLNDrONCz9aNavH/nffsvu0mlYXn2VxNNPJ3XcVc40LFV849gYU3fsP800anFdupD25z/R7f2ltJ1wOwU//ED2jTfxzdCh5L7wIiUHDtR3iKaGjtZp5KN9P5JSxcXFnHLKKRH7MiJYIjFHCW9KCq2uuYaui96h/ZRH8bZowbZJk9g4YCDbHplMYU7gV5hMQ3W0JpJw1OZ+JKUef/zxiA+esERijiri85F8wQV0mTOHzrNfotlPzyJ3+nSyBg0m+3e/4+AXX9R3iKYadj+SurkfCUB2djZvvvkm1157bQ1/CzVj10jMUSuhd2869O5NYU4OuS++yJ6X57L/7YXEn5xBq3HjaD54MBJj/wJV2frAA+Svr9tBDHE9TiDtD3+odLndj6Tu7kdy66238sgjj9TJRJZVsf8ic9TztWvHMRMm0ObGG9kz/1VyZ85g823/j5j0dFKvvIIWv/gF3pSU+g7TBGH3I6n9/UjeeOMN2rZtS58+fXj//fdDPTy1YonENBmeZs1IHXMlLa8YTd77zjQs2//6KDum/YMWwy+h5dixxHWxaVj8VXXmEA12P5La349k2bJlLFiwgLfeeovDhw+zb98+xowZwwsvvFDpOmvLrpGYJkc8HpqfM5Bjn3+OLq/OJ3nIEPbMfYVNQy/gx1/fwIH//pemMONDQ2X3I6mb+5E8+OCDZGdn89133zF79mzOOeeciCQRsERimrj4E06g3YMP0PW9d2l9000cWrOGH67+Jd9eMpw98/5DScCkfyby/O9Hsnjx4rL7kfTq1YuRI0eyf/9+vvzyS/r27Uvv3r25//77uesu567cpfcjqepieyhuvPFGpk+fTr9+/diwYUPQ+5FMmTKFZ555hvXr1wddxw033EBeXh4ZGRk88sgjQe9HkpGRQb9+/fif35dpS+9H8vjjj/PYY48Bzv1IJk+ezCmnnFJ2sT2YnJwcLrjggrD2vTZsri1j/JTk57PvjTfInT6D/A0b8LZqRctRo2g5ehQx9XCPiPpgc23VH7sfiTFHAWcalhF0ee1VOj33LAk9e7Jz2jRnGpY//JHDX39d3yEa0+DYxXZjghARmp1xBs3OOMOdhmUme+a/yt7//IfEfv2caVh+9jObhqUBs/uRRI91bRkTouI9e9g9dy67X5xF0datxB57LC2vGkuLSy7B4/ahHw2sa6tpsq4tY6LA26IFra+7jq6LF9Hu0b/iSUlh218msXHgOWz/618p3LKlvkOsM03hA6Y5ItzftyUSY2pIfD5SLryQLi/P4diXZtHszDPZ9exzZJ03iM233cah1avrO8SwxMfHs2vXLksmTYSqsmvXrnLflakp69oypg4Ubt5M7ouz2DN3LiX795Nw8smkjh9H80GDGt00LIWFhWRnZ3P48OH6DsVESXx8PB06dMDn85UrD7VryxKJMXWoOO8Ae+fPJ3fmTAp/+MGZhmXMlc40LMnJ9R2eMTViicSPJRITbVpcTN4HH5D7/HQOLl+OJCbSYvhwUseOIbZz5/oOz5iQWCLxY4nE1KfD69eTO30G+958Ey0qImnAAFLHjSPx9L5VzqVkTH2zROLHEolpCIp27GD3S7PZPXs2xbm5xJ1wAi0uvZTYYzsRk5aOr1063ubN6ztMY8pYIvFjicQ0JCX5+ex7/XVnGpaNG8st8zRrRkx6Gr60dHzpacSk+T93fnoSEuopctPUhJpIGtdwEmOOAp64OFqMHEnKiBEUbd1K4ZatFG7JOfJ86xaKtmzl8Pr1FO/aVaG9NyWFmPR0fGlp5ZKOLz3dKW/bFomNrYc9M02VJRJj6omI4EtPx5eeDpwStE5Jfj5F27ZRuGUrRVu3lEs0hVu2cPCLLyjZuzdwxXhbt3ISTECyiUlzE06bNojXG/mdNE1CRBOJiAwBHge8wDOq+lCQOpcB9wAKrFbVK9zyccBdbrVJqjrdLe8DPA8kAG8Bv9Wm0D9nmiRPXByxnToR26lTpXVKDhygcNs2CnO2VEg2+d98Q96yZejBg+Ubeb3EtG2LLy3Nr9usfBeaNzXVBgOYkETsGomIeIENwCAgG/gMGK2q6/zqdANeBs5R1d0i0lZVt4tIKrACyMRJMCuBPm6d5cBvgU9wEskTqvp2VbHYNRLTlKkqJfv2UbjVOYsp7UI7knS2UrRlC1pYWK6dxMa612jSyiWY0rMaX1oanuRkSzZHsYZwjaQvkKWqm9yAZgMXA+v86lwHTFPV3QCqut0tPx9YrKq5btvFwBAReR9IVtX/uuUzgEuAKhOJMU2ZiOBNScGbkkL8T34StI6WlFCcm1u+62zrkWRzYPlnFG3fDu6dAsvWnZjoJJq0NGLapVfoQvOlpeFJTIzGbpp6FMlE0h740e91NnB6QJ3uACKyDKf76x5VXVhJ2/buIztIuTEmDOLxENO6NTGtW5PQq2fQOlpURNHOneXOavyTzuH3N1C8c2eFdp6UlCPJJr3iKLSYtDQ8NjigUYtkIgl2vhvYjxYDdAMGAB2Aj0SkZxVtQ1mns3GR64HrATpV0b9sjAmNxMSUJYTKlBQUuIMDgiebQ6tWURw4OADwtmrlrLtdupNgArvT2rRpdHOWNSWR/M1kAx39XncAcoLU+URVC4FvReRrnMSSjZNc/Nu+75Z3qGadAKjqU8BT4Fwjqe1OGGNC54mNJbZjR2I7dqy0TsnBgxRu3VZ+YICbdPI3fcuBZR9TEjg4wOMpGxwQbBSaLy0Nb6tWdqOxehLJRPIZ0E1EugCbgVHAFQF1XgVGA8+LSGucrq5NwDfAAyLS0q03GLhTVXNFZL+I9AM+Ba4C/hbBfTDG1DFPYiJxx3Uh7rguQZerKiX79wcf8rx1K4fXrSPv3ffQgoJy7cTnI+aYY9zv0wRPNp6UFBscEAERSySqWiQiNwPv4Fz/eFZVvxKR+4AVqrrAXTZYRNYBxcAEVd0FICJ/wUlGAPeVXngHbuDI8N+3sQvtxhxVRARvcrIzW/JPugeto6oU795dcRRazhYKt27l4IoVFG0LMjggIaF8t1mQ6zbepKPnbpfRYlOkGGOOSlpcTNHOnRRt2eIOfa445Llo504IeA/0JCcH70LzHxwQF1dPexVddTb8V0SOB7JVNV9EBgAZwAxV3RN+mMYYExni9eI75hh8xxxDZbOTaUEBhdt3BHShHUk2h1evoXhPxbc6b2qqm2wqDgzwpacT07ZtkxocEMqezgMyRaQr8G9gATALuCCSgRljTKRJbCyxHdoT26HybxGUHDrkfqem4ii0wh++5+Cnn1KSl1e+kcdDTJs2lSabmLQ0Ylq3PmoGB4SSSErc6x3Dgamq+jcR+SLSgRljTEPgSUggrksX4roEHxwAUJyX59eFtsW5duMmm/z168lbuhTNzy/fyOfD17ZtkIEB7cqee1u0aBSDA0JJJIUiMhoYB/zcLfNVUd8YY5oUb1IS3m7diOvWLehyVaV4z55yycb/DOfQF1+wb+E2KCoq107i46sc8hyTno43KSkau1ilUBLJ1cCvgftV9Vt3OO8LkQ3LGGOOHiJCTMuWxLRsSfyJJwatoyUlzuCArVvd0Wd+XWhbt3Bg2TKKduyoODggKSn4KLR2bpda+/aIL7Kf/Ws0asv9XkdHVV2B7/GUAAAZL0lEQVQTuZDqno3aMsYcDbSwkKLt2ysdhVa4dSvFubnl2nRZ8Brx3YMPo65OXY7aeh8Y5tZdBewQkQ9U9bZaRWaMMaZWxOfD1749vvZVDA7Iz/e7YdqWKmcZqCuhdG2lqOo+EbkWeE5V7xaRRnVGYowxTYUnLo7YY48l9thjo7fNEOrEiEg6cBnwRoTjMcYY08iEkkjuw5nK5BtV/UxEjgM2RjYsY4wxjUW1XVuqOheY6/d6EzAikkEZY4xpPKo9IxGRDiIyX0S2i8g2EZknIh2qa2eMMaZpCKVr6zmcaVHa4dyN8HW3zBhjjAkpkbRR1edUtch9PA+0iXBcxhhjGolQEslOERkjIl73MQbYFenAjDHGNA6hJJJf4gz93QpsAUbiTJtijDHGVJ9IVPUHVR2mqm1Uta2qXgJcGoXYjDHGNAK1nQzfpkcxxhgD1D6RNPwJ8o0xxkRFbRPJ0X+jd2OMMSGp9JvtIrKf4AlDoNJbIBtjjGliKk0kqto8moEYY4xpnI6OO88bY4ypN5ZIjDHGhMUSiTHGmLBYIjHGGBOWUKaR3y8i+wIeP7pTyx9XTdshIvK1iGSJyB1Blo8XkR0issp9XOu37GERWes+Lvcrf15EvvVr07umO22MMabuhHLP9ilADjALZ+jvKCAN+Bp4FhgQrJGIeIFpwCAgG/hMRBao6rqAqnNU9eaAthcCpwK9gTjgAxF5W1X3uVUmqOorIcRujDEmwkLp2hqiqv9S1f2quk9VnwIuUNU5QMsq2vUFslR1k6oWALOBi0OM60TgA3fa+gPAamBIiG2NMcZEUSiJpERELhMRj/u4zG9ZVd9wbw/86Pc62y0LNEJE1ojIKyLS0S1bDQwVkUQRaQ0MBDr6tbnfbfOYiMSFsA/GGGMiJJREciUwFtjuPsYCY0QkAbi5inbB5uMKTDyvA51VNQNYAkwHUNVFwFvAx8BLwH+BIrfNncAJwGlAKjAx6MZFrheRFSKyYseOHdXtozHGmFoKZRr5Tar6c1Vt7T5+rqpZqnpIVf+viqbZlD+L6IBzrcV/3btUNd99+TTQx2/Z/araW1UH4SSljW75FnXk49zyt28lcT+lqpmqmtmmjd3Q0RhjIiWUUVsd3BFa20Vkm4jME5EOIaz7M6CbiHQRkVici/QLAtad7vdyGLDeLfeKSCv3eQaQASzybyMiAlwCrA0hFmOMMRESyqit53BGbP3CfT3GLRtUVSNVLRKRm4F3AC/wrKp+JSL3AStUdQFwi4gMw+m2ygXGu819wEdOrmAfMEZVS7u2XhSRNjhnKauAX4eyo8YYYyJDVKueEV5EVqlq7+rKGrLMzExdsWJFfYdhjDGNioisVNXM6uqFcrF9p4iMcbubvCIyBtgVfojGGGOOBqEkkl8ClwFbgS3ASODqSAZljDGm8Qhl1NYPqjpMVduoaltVvQS4NAqxGWOMaQRCudgezG3A1LoMxESPqqLqfKlHVSlRUNwyv+clqm6d8m1Kyp5X0sa97FYSpA2423PblJSEuG33uf+2UcrFXtoGDXHb7rrLxRpsPZRvg1tWUlI+LqBC7EfKK2677ECJIM4PBEEEPALuYJOK5e5zZ5m4Zc5zcZ9TVn6kTNx1ltUtKwNP2baOlHv8nuO3Hk/pdtz1E1C3bJsVtn9kHwgor7hfzmuPJ8R98N9mkH0gIJayfXDjD9zfKvfBLw7jqG0iaRJH8K/vfM3q7D0hvMmUvhkDAW/MQd+sCOHNvPTNqpI2R2IKLSmUf0M1xtSFYMkQ/6SHlH0oOJLQwOOpmAwpqxs8AQeWe8R/W1IuQeJX9q+xfeiYmhjR41DbRNIk3o4OFRZzIL+o3C/J/w8hpuwT1pFPc56AX6z/p8Ryn+T8llf8RFldG/9PT+XbeCT4tst/Oiv/CdD/068nSLzB9s3jrlSCtCFI7IGfeD3lYgv1GFXSxnPkuFTaptz+Vv4Jt0KbauINfAPxBGlT2XEtPeaBHwb8z4Cq/EBQSXnp2Vq5szi/7UCQDyiU/wATuH3/M65gZ434x1FSfh2BZ47lt+m/j0HK/dbj/6GsfBxH2hFY7rfNsg955fbRf1sVt1nxw5r/h8mKZ6sVj3/58orHP2CbVZUH/K5KguxzsN9LjDfyn/srTSQisp/gCUOAhIhF1ID86aIT6zsE0wSIX8JpIif75ihTaSJR1ebRDMQYY0zjZHdINMYYExZLJMYYY8JiicQYY0xYLJEYY4wJiyUSY4wxYbFEYowxJiyWSIwxxoTFEokxxpiwWCIxxhgTFkskxhhjwmKJxBhjTFgskRhjjAmLJRJjjDFhsURijDEmLJZIjDHGhMUSiTHGmLBYIjHGGBMWSyTGGGPCEtFEIiJDRORrEckSkTuCLB8vIjtEZJX7uNZv2cMistZ9XO5X3kVEPhWRjSIyR0RiI7kPxhhjqhaxRCIiXmAaMBQ4ERgtIicGqTpHVXu7j2fcthcCpwK9gdOBCSKS7NZ/GHhMVbsBu4FrIrUPxhhjqhfJM5K+QJaqblLVAmA2cHGIbU8EPlDVIlU9AKwGhoiIAOcAr7j1pgOX1HHcxhhjaiCSiaQ98KPf62y3LNAIEVkjIq+ISEe3bDUwVEQSRaQ1MBDoCLQC9qhqUTXrNMYYEyWRTCQSpEwDXr8OdFbVDGAJzhkGqroIeAv4GHgJ+C9QFOI6nY2LXC8iK0RkxY4dO2q3B8YYY6oVyUSSjXMWUaoDkONfQVV3qWq++/JpoI/fsvvd6yaDcBLIRmAn0EJEYipbp1/7p1Q1U1Uz27RpUyc7ZIwxpqJIJpLPgG7uKKtYYBSwwL+CiKT7vRwGrHfLvSLSyn2eAWQAi1RVgaXASLfNOOC1CO6DMcaYasRUX6V2VLVIRG4G3gG8wLOq+pWI3AesUNUFwC0iMgyn2yoXGO829wEfOdfW2QeM8bsuMhGYLSKTgC+Af0dqH4wxxlRPnA/5R7fMzExdsWJFfYdhjDGNioisVNXM6urZN9uNMcaExRKJMcaYsFgiMcYYExZLJMYYY8JiicQYY0xYLJEYY4wJiyUSY4wxYbFEYowxJiyWSIwxxoTFEokxxpiwWCIxxhgTFkskxhhjwmKJxBhjTFgskRhjjAmLJRJjjDFhsURijDEmLJZIjDHGhMUSiTHGmLBYIjHGGBMWSyTGGGPCYonEGGNMWGLqO4CGbNInk1i7cy3JsckkxyXTPLY5ybFHfpY9/JYlxSbh8/jqO3RjjIkaSyRVaJPQhhbxLdhfsJ8tB7awv2A/+wr2UVhSWGW7xJjEkBOP/7Lmsc1JiElARKK0h8YYEz5LJFX41cm/qlCmqhwuPuwklfx97C90fu4rOPIoW+Ymnpy8HL4u+Jp9Bfs4UHigym3GeGLKJZ1yCSeuYuJJjksm2eckpyRfEl6PN1KHwxhjgrJEUkMiQkJMAgkxCbRNbFvj9kUlReQV5JUlmb0Fe8uelyagsuduYsrOyy5bVqRFVa4/yZdUPvH43GQTkJRS4lIqlMXHxNf2sBhjmrCIJhIRGQI8DniBZ1T1oYDl44HJwGa36O+q+oy77BHgQpwBAYuB36qqisj7QDpwyG0zWFW3R3I/6lKMJ4YW8S1oEd+ixm1VlUNFhyqe+RQGT0D78vfxw/4f2L/LKTtUdKjK9cd6Yo+c5QSeDcUmkxKXUqGstH6SLwmP2NgNY5qiiCUSEfEC04BBQDbwmYgsUNV1AVXnqOrNAW3PBM4CMtyi/wN+Brzvvr5SVVdEKvaGSkRI9CWS6EskrVlajdsXlhSyv2B/pWc+pcmntGz34d18v+/7sjbFWlx5bAhJsUnVXgsq65oL6KaL9caGc2iMMfUokmckfYEsVd0EICKzgYuBwEQSjALxQCwggA/YFqE4mwyfx0dqfCqp8ak1bquqHCg8UCHxBEtKpT837dlU9vxw8eEq1x/vjQ96LSjoNaGAa0jNfM1sgIIx9SiSiaQ98KPf62zg9CD1RojI2cAG4Heq+qOq/ldElgJbcBLJ31V1vV+b50SkGJgHTFJVjcwumFIizhlHUmwS6aTXuH1BcUH5M6D8ionHPzHtOLiDb/Z8U3Y2pFT+K/aIp2LCqSLxBCYsG65tTHgimUiCfUQMfDd4HXhJVfNF5NfAdOAcEekK9AA6uPUWi8jZqvohTrfWZhFpjpNIxgIzKmxc5HrgeoBOnTrVyQ6Z2ov1xtI6oTWtE1rXuG2JlpBXmFdhNFxp4tmbv7dCUtp2cFtZ/YKSgirXnxCTUHGYdiXXiQK75Gy4tjGRTSTZQEe/1x2AHP8KqrrL7+XTwMPu8+HAJ6qaByAibwP9gA9VdbPbdr+IzMLpQquQSFT1KeApgMzMTDtjacQ84il7426f1L7G7fOL8yskoMCzIv8ktPXgVjbs3uCcDRXur3LdMRJT5QCFwG66lNjyAxZsuLY5GkQykXwGdBORLjijskYBV/hXEJF0Vd3ivhwGlHZf/QBcJyIP4pzZ/AyYKiIxQAtV3SkiPuAiYEkE98EcBeK8cbRJbEObxDY1bltcUkxeYV7114Ty97GvcB/78/eTk5dTVr+opOrh2gkxCcR744mLiSPOG/CIiSPO4/yM98YT642tULesPMZvuV/7YO1sdJ2paxFLJKpaJCI3A+/gDP99VlW/EpH7gBWqugC4RUSGAUVALjDebf4KcA7wJU532EJVfV1EmgHvuEnEi5NEno7UPhjj9XhJiUshJS6lxm1Lv7xa2WCEfQX7yCvII784/8ij6MjzffnOIIWC4gIOFx0uV69ES2q9Tz6Pr1wCCprA/F7XJElV1s7n8VkX4FFMmsJ16szMTF2xosmNFjZHKVWlqKSI/OJ8DhcfDpqESh+BCci/TmVJ6nCRWx6w7uquNVVFkOBJqhZnVIHtgiXC0uUxHvvOdThEZKWqZlZXz46yMY2MiODz+vB5fSSRFLXtlmgJBcUF5ZJLWTIqPhx6kiqpWJ5XmEf+oYpJsKC4oNrZHKoSIzGVdhv6J6lgSSk+Jp5YT5BkFROQzIKckTW1sy9LJMaYkHjEQ3xMfNSn0ik9+6pJ8gp2RhWs3YHCA5XWCUesJzasa1uBCa2q7sfStjGemHpLYJZIjDENWownhhhPDM18zaK2TVWlsKSwZmdaQZJXsDqHig6xN39v0HVXNzijKh7xBL1G9fiAx+mY3LH6FYTBEokxxgQQEWK9sc7UPVGcvae4pDjoda7qklRV18OiMf2QJRJjjGkgvB4viR5nPr3GxAaUG2OMCYslEmOMMWGxRGKMMSYslkiMMcaExRKJMcaYsFgiMcYYExZLJMYYY8Ji3yMxxphQqEJJMWhxwM+Siq8rXVYMJe7yqpZVWF+wulVt2289/W6AZjW/oVxNWCIx5mgRzptQuTrB1lPVG2AV6w/lzbKsTl3E6beuat+giytJDpXUbYzECxmXWyKpV0vuhezPIruNWk3jX4s20dhOg92XBnq8IOCNL8xPqI2ReMHjDfjpcX6Kp2JZSHW9IL4a1A187VcedFmwWCqrW8n6xVP9PpWrW02cFeq664kSSyRVKSly/klrqsYzcNaifk23UdtJQSO+L7XZRi22E41t1Go7EvAGUsmbVNA3EAntjajGb3SeKtbTON/oTGRZIqnK4L/UdwTGGNPg2UcCY4wxYbFEYowxJiyWSIwxxoTFEokxxpiwWCIxxhgTFkskxhhjwmKJxBhjTFgskRhjjAmLaK2mgmhcRGQH8H0tm7cGdtZhOHXF4qoZi6tmLK6aOVrjOlZV21RXqUkkknCIyApVzazvOAJZXDVjcdWMxVUzTT0u69oyxhgTFkskxhhjwmKJpHpP1XcAlbC4asbiqhmLq2aadFx2jcQYY0xY7IzEGGNMWJpsIhGRjiKyVETWi8hXIvLbIHVERJ4QkSwRWSMip/otGyciG93HuCjHdaUbzxoR+VhETvZb9p2IfCkiq0RkRZTjGiAie91trxKRP/stGyIiX7vH8o4oxzXBL6a1IlIsIqnuskgdr3gRWS4iq9247g1SJ05E5rjH5FMR6ey37E63/GsROT/Kcd0mIuvcv693ReRYv2XFfsdyQZTjGi8iO/y2f63fskj9P4YS12N+MW0QkT1+yyJyvPzW7xWRL0TkjSDLovf3papN8gGkA6e6z5sDG4ATA+pcALyNc6u8fsCnbnkqsMn92dJ93jKKcZ1Zuj1gaGlc7uvvgNb1dLwGAG8EaesFvgGOA2KB1YFtIxlXQP2fA+9F4XgJkOQ+9wGfAv0C6twI/NN9PgqY4z4/0T1GcUAX99h5oxjXQCDRfX5DaVzu67y6PlY1iGs88PcgbSP5/1htXAH1fwM8G+nj5bf+24BZlfzfRe3vq8mekajqFlX93H2+H1gPtA+odjEwQx2fAC1EJB04H1isqrmquhtYDAyJVlyq+rG7XYBPgA51se1w46pCXyBLVTepagEwG+fY1kdco4GX6mLb1cSlqprnvvS5j8ALkhcD093nrwDnioi45bNVNV9VvwWycI5hVOJS1aWqetB9Ga2/r1COV2Ui+f9Y07ii8vcFICIdgAuBZyqpErW/ryabSPy5p3yn4Hza8Nce+NHvdbZbVll5tOLydw3OWVMpBRaJyEoRub6uYwohrjPcboC3ReQkt6xBHC8RScR5g5nnVxyx4+V2O6wCtuO80VX696WqRcBeoBURPl4hxOUv8O8rXkRWiMgnInJJXcVUg7hGuF1ur4hIR7esQRwvtwuwC/CeX3HEjhcwFfg9UFLJ8qj9fTX5RCIiSThvLLeq6r7AxUGaaBXl0YqrtM5AnH/0iX7FZ6nqqThdXjeJyNlRjOtznCkVTgb+Brxa2izIqqJ+vHC6tZapaq5fWcSOl6oWq2pvnE/0fUWkZ2DYwZpVUR6tuJzgRMYAmcBkv+JO6nxT+gpgqogcH8W4Xgc6q2oGsIQjn7YbxPHC6T56RVWL/coicrxE5CJgu6qurKpakLKI/H016UQiIj6cN58XVfU/QapkAx39XncAcqooj1ZciEgGzintxaq6q7RcVXPcn9uB+dRRl0gocanqvtJuAFV9C/CJSGsawPFyjSKg2yGSx8tvG3uA96nY3VJ2XEQkBkgBconw8QohLkTkPOCPwDBVzfdrU3q8NrltT4lWXKq6yy+Wp4E+7vN6P16uqv6+6vp4nQUME5HvcLqKzxGRFwLqRO/vK5wLLI35gZOVZwBTq6hzIeUvti93y1OBb3Eu7LV0n6dGMa5OOP2aZwaUNwOa+z3/GBgSxbjSOPLdpL7AD267GJwLoF04crH9pGjF5dYr/SdqFqXj1QZo4T5PAD4CLgqocxPlL4a+7D4/ifIXQzdRdxfbQ4nrFJwLsN0CylsCce7z1sBG6m7QRChxpfs9Hw584j6P5P9jtXG5y36CM3BDonG8ArY9gOAX26P29xVD03UWMBb40u3/BPgDzps0qvpP4C2ckVtZwEHgandZroj8BfjMbXeflu8uiXRcf8bp6/yHc+2MInVOn48B5rtlMcAsVV0YxbhGAjeISBFwCBilzl9ukYjcDLyDM4LrWVX9KopxgfPGs0hVD/i1jeTxSgemi4gX58z/ZVV9Q0TuA1ao6gLg38BMEcnCSXKj3Ji/EpGXgXVAEXCTlu8uiXRck4EkYK57bH5Q1WFAD+BfIlLitn1IVddFMa5bRGQYzjHJxRnFFen/x1DiAuci+2z3771UJI9XUPX192XfbDfGGBOWJn2NxBhjTPgskRhjjAmLJRJjjDFhsURijDEmLJZIjDHGhMUSiTHGmLBYIjGmgRBnSvvWtWw7XkTa1cW6jKkpSyTGHB3GA+2qq2RMJFgiMSaAiHQWkf+JyDPi3AjrRRE5T0SWiXPjpL7u42P3pkIfi8hP3La3iciz7vNebvvESrbTSkQWuev4F36T6YnIGHFuqLRKRP7lfrMaEckTkUdF5HNxbjrVRkRG4kyu+KJbP8FdzW/cel+KyAmRPGamabNEYkxwXYHHgQzgBJzZW38K3I4zBcv/gLNV9RScKWsecNtNBbqKyHDgOeBXeuTeHoHuBv7PXccC3GldRKQHcDnOzMS9gWLgSrdNM+BzdWYs/gC4W1VfAVYAV6pqb1U95Nbd6dZ70o3bmIhoynNtGVOVb1X1SwAR+Qp4V1VVRL4EOuNMAjldRLrhTMHtA1DVEhEZD6wB/qWqy6rYxtnApW67N0Wk9GZl5+LMbPuZO9dVAs69MMC598Qc9/kLQFWzHZcuW1m6HWMiwRKJMcHl+z0v8XtdgvN/8xdgqaoOF+eGWu/71e8G5BHaNYtgk90JMF1V76xl+1KlMRdj/+smgqxry5jaSQE2u8/HlxaKSApOl9jZQCv3+kVlPsTtshKRoTjTjgO8C4wUkbbuslT37nvg/M+WrvMK4P/c5/tx7llvTNRZIjGmdh4BHhSRZThT45d6DPiHqm7AuXvlQ6UJIYh7gbNF5HNgMM79W3CnGr8L5xbAa3DuQZ7utjkAnCQiK4FzgPvc8ueBfwZcbDcmKmwaeWMaERHJU9Wk+o7DGH92RmKMMSYsdkZiTISJyNXAbwOKl6nqTfURjzF1zRKJMcaYsFjXljHGmLBYIjHGGBMWSyTGGGPCYonEGGNMWCyRGGOMCcv/B58WH6AaoHYiAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x165913862e8>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# summarize results\n",
    "print(\"Best: %f using %s\" % (gsearch2_1.best_score_, gsearch2_1.best_params_))\n",
    "test_means = gsearch2_1.cv_results_[ 'mean_test_score' ]\n",
    "test_stds = gsearch2_1.cv_results_[ 'std_test_score' ]\n",
    "train_means = gsearch2_1.cv_results_[ 'mean_train_score' ]\n",
    "train_stds = gsearch2_1.cv_results_[ 'std_train_score' ]\n",
    "\n",
    "pd.DataFrame(gsearch2_1.cv_results_).to_csv('my_preds_maxdepth_min_child_weights_1.csv')\n",
    "\n",
    "# plot results\n",
    "test_scores = np.array(test_means).reshape(len(max_depth), len(min_child_weight))\n",
    "train_scores = np.array(train_means).reshape(len(max_depth), len(min_child_weight))\n",
    "\n",
    "for i, value in enumerate(max_depth):\n",
    "    pyplot.plot(min_child_weight, -test_scores[i], label= 'test_max_depth:'   + str(value))\n",
    "#for i, value in enumerate(min_child_weight):\n",
    "#    pyplot.plot(max_depth, train_scores[i], label= 'train_min_child_weight:'   + str(value))\n",
    "    \n",
    "pyplot.legend()\n",
    "pyplot.xlabel( 'max_depth' )                                                                                                      \n",
    "pyplot.ylabel( 'Log Loss' )\n",
    "pyplot.savefig('max_depth_vs_min_child_weght_1.png' )"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### max_depth': 5 ，min_child_weight': 1 下结果最优 -0.582154"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 70,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'max_depth': [4, 3, 2], 'min_child_weight': [1, 2, 3]}"
      ]
     },
     "execution_count": 70,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#max_depth 建议5以下，取4，3，2， min_child_weight=1／sqrt(ratio_rare_event) =5.5\n",
    "max_depth = [4,3,2]\n",
    "min_child_weight = [1,2,3]\n",
    "param_test2_2 = dict(max_depth=max_depth, min_child_weight=min_child_weight)\n",
    "param_test2_2"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 71,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "D:\\anaconda\\lib\\site-packages\\sklearn\\model_selection\\_search.py:761: DeprecationWarning: The grid_scores_ attribute was deprecated in version 0.18 in favor of the more elaborate cv_results_ attribute. The grid_scores_ attribute will not be available from 0.20\n",
      "  DeprecationWarning)\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "([mean: -0.58523, std: 0.00338, params: {'max_depth': 4, 'min_child_weight': 1},\n",
       "  mean: -0.58561, std: 0.00381, params: {'max_depth': 4, 'min_child_weight': 2},\n",
       "  mean: -0.58576, std: 0.00347, params: {'max_depth': 4, 'min_child_weight': 3},\n",
       "  mean: -0.59352, std: 0.00344, params: {'max_depth': 3, 'min_child_weight': 1},\n",
       "  mean: -0.59361, std: 0.00339, params: {'max_depth': 3, 'min_child_weight': 2},\n",
       "  mean: -0.59364, std: 0.00321, params: {'max_depth': 3, 'min_child_weight': 3},\n",
       "  mean: -0.60801, std: 0.00334, params: {'max_depth': 2, 'min_child_weight': 1},\n",
       "  mean: -0.60799, std: 0.00344, params: {'max_depth': 2, 'min_child_weight': 2},\n",
       "  mean: -0.60800, std: 0.00328, params: {'max_depth': 2, 'min_child_weight': 3}],\n",
       " {'max_depth': 4, 'min_child_weight': 1},\n",
       " -0.5852302830815046)"
      ]
     },
     "execution_count": 71,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "xgb2_2 = XGBClassifier(\n",
    "        learning_rate =0.1,\n",
    "        n_estimators=340,  #第一轮参数调整得到的n_estimators最优值\n",
    "        max_depth=4,\n",
    "        min_child_weight=1,\n",
    "        gamma=0,\n",
    "        subsample=0.7,\n",
    "        colsample_bytree=0.8,\n",
    "        colsample_bylevel = 0.7,\n",
    "        objective= 'multi:softprob',\n",
    "        seed=3)\n",
    "\n",
    "\n",
    "gsearch2_2 = GridSearchCV(xgb2_2, param_grid = param_test2_2, scoring='neg_log_loss',n_jobs=-1, cv=kfold)\n",
    "gsearch2_2.fit(X_train , y_train)\n",
    "\n",
    "gsearch2_2.grid_scores_, gsearch2_2.best_params_,     gsearch2_2.best_score_"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 72,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "D:\\anaconda\\lib\\site-packages\\sklearn\\utils\\deprecation.py:122: FutureWarning: You are accessing a training score ('mean_train_score'), which will not be available by default any more in 0.21. If you need training scores, please set return_train_score=True\n",
      "  warnings.warn(*warn_args, **warn_kwargs)\n",
      "D:\\anaconda\\lib\\site-packages\\sklearn\\utils\\deprecation.py:122: FutureWarning: You are accessing a training score ('split0_train_score'), which will not be available by default any more in 0.21. If you need training scores, please set return_train_score=True\n",
      "  warnings.warn(*warn_args, **warn_kwargs)\n",
      "D:\\anaconda\\lib\\site-packages\\sklearn\\utils\\deprecation.py:122: FutureWarning: You are accessing a training score ('split1_train_score'), which will not be available by default any more in 0.21. If you need training scores, please set return_train_score=True\n",
      "  warnings.warn(*warn_args, **warn_kwargs)\n",
      "D:\\anaconda\\lib\\site-packages\\sklearn\\utils\\deprecation.py:122: FutureWarning: You are accessing a training score ('split2_train_score'), which will not be available by default any more in 0.21. If you need training scores, please set return_train_score=True\n",
      "  warnings.warn(*warn_args, **warn_kwargs)\n",
      "D:\\anaconda\\lib\\site-packages\\sklearn\\utils\\deprecation.py:122: FutureWarning: You are accessing a training score ('split3_train_score'), which will not be available by default any more in 0.21. If you need training scores, please set return_train_score=True\n",
      "  warnings.warn(*warn_args, **warn_kwargs)\n",
      "D:\\anaconda\\lib\\site-packages\\sklearn\\utils\\deprecation.py:122: FutureWarning: You are accessing a training score ('split4_train_score'), which will not be available by default any more in 0.21. If you need training scores, please set return_train_score=True\n",
      "  warnings.warn(*warn_args, **warn_kwargs)\n",
      "D:\\anaconda\\lib\\site-packages\\sklearn\\utils\\deprecation.py:122: FutureWarning: You are accessing a training score ('std_train_score'), which will not be available by default any more in 0.21. If you need training scores, please set return_train_score=True\n",
      "  warnings.warn(*warn_args, **warn_kwargs)\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "{'mean_fit_time': array([ 472.92665515,  467.57786527,  451.77480674,  358.47235394,\n",
       "         395.5849802 ,  570.74175348,  432.23635201,  404.27698607,\n",
       "         332.64838967]),\n",
       " 'mean_score_time': array([ 0.82318926,  0.84214358,  0.76914496,  0.96456409,  1.61905918,\n",
       "         1.178755  ,  0.88284531,  0.80635915,  0.5510715 ]),\n",
       " 'mean_test_score': array([-0.58523028, -0.58560532, -0.58575892, -0.59352338, -0.59361177,\n",
       "        -0.59363606, -0.60800785, -0.60799416, -0.60800213]),\n",
       " 'mean_train_score': array([-0.52258242, -0.52449693, -0.52573513, -0.56172457, -0.56231698,\n",
       "        -0.56290576, -0.59442558, -0.59452679, -0.59468354]),\n",
       " 'param_max_depth': masked_array(data = [4 4 4 3 3 3 2 2 2],\n",
       "              mask = [False False False False False False False False False],\n",
       "        fill_value = ?),\n",
       " 'param_min_child_weight': masked_array(data = [1 2 3 1 2 3 1 2 3],\n",
       "              mask = [False False False False False False False False False],\n",
       "        fill_value = ?),\n",
       " 'params': [{'max_depth': 4, 'min_child_weight': 1},\n",
       "  {'max_depth': 4, 'min_child_weight': 2},\n",
       "  {'max_depth': 4, 'min_child_weight': 3},\n",
       "  {'max_depth': 3, 'min_child_weight': 1},\n",
       "  {'max_depth': 3, 'min_child_weight': 2},\n",
       "  {'max_depth': 3, 'min_child_weight': 3},\n",
       "  {'max_depth': 2, 'min_child_weight': 1},\n",
       "  {'max_depth': 2, 'min_child_weight': 2},\n",
       "  {'max_depth': 2, 'min_child_weight': 3}],\n",
       " 'rank_test_score': array([1, 2, 3, 4, 5, 6, 9, 7, 8]),\n",
       " 'split0_test_score': array([-0.5799731 , -0.58001046, -0.58020518, -0.58798653, -0.58793865,\n",
       "        -0.58817796, -0.60272494, -0.60251131, -0.60278658]),\n",
       " 'split0_train_score': array([-0.52355827, -0.52575956, -0.5270894 , -0.56307002, -0.56351022,\n",
       "        -0.56438846, -0.59588184, -0.59592249, -0.5960879 ]),\n",
       " 'split1_test_score': array([-0.58359141, -0.58290924, -0.58375175, -0.59199525, -0.59245013,\n",
       "        -0.59244943, -0.60818748, -0.60795353, -0.60781702]),\n",
       " 'split1_train_score': array([-0.52211939, -0.52407937, -0.52528526, -0.56172755, -0.56263937,\n",
       "        -0.5632706 , -0.59446395, -0.59439616, -0.59465101]),\n",
       " 'split2_test_score': array([-0.58504328, -0.5857417 , -0.58628472, -0.59336152, -0.59368958,\n",
       "        -0.59393675, -0.60695711, -0.60734591, -0.6072559 ]),\n",
       " 'split2_train_score': array([-0.52309347, -0.52484688, -0.5257016 , -0.56132676, -0.5619144 ,\n",
       "        -0.56278419, -0.59428046, -0.5945013 , -0.59459987]),\n",
       " 'split3_test_score': array([-0.58965195, -0.58900492, -0.58882038, -0.59744694, -0.59778248,\n",
       "        -0.59702027, -0.61307786, -0.61327153, -0.61294265]),\n",
       " 'split3_train_score': array([-0.52256747, -0.52398977, -0.5257181 , -0.56147266, -0.56200901,\n",
       "        -0.56203818, -0.59361422, -0.59382212, -0.59388808]),\n",
       " 'split4_test_score': array([-0.58789248, -0.59036173, -0.58973378, -0.59682769, -0.59619881,\n",
       "        -0.59659682, -0.60909221, -0.60888876, -0.60920888]),\n",
       " 'split4_train_score': array([-0.52157351, -0.52380906, -0.52488129, -0.56102586, -0.56151189,\n",
       "        -0.56204738, -0.59388742, -0.59399188, -0.59419084]),\n",
       " 'std_fit_time': array([  0.85756766,  12.13050363,   3.62124297,  11.57720142,\n",
       "         70.87024587,  10.30115183,  17.16706032,  26.52272617,  20.46537728]),\n",
       " 'std_score_time': array([ 0.08187607,  0.09045533,  0.02766949,  0.41828917,  0.46160116,\n",
       "         0.10570296,  0.13084101,  0.05918518,  0.12002779]),\n",
       " 'std_test_score': array([ 0.00337635,  0.00381492,  0.00347333,  0.00344474,  0.00339496,\n",
       "         0.00320963,  0.00334453,  0.00343912,  0.003276  ]),\n",
       " 'std_train_score': array([ 0.00069943,  0.00072408,  0.00074389,  0.00070995,  0.00069755,\n",
       "         0.00087588,  0.00078628,  0.0007414 ,  0.00075575])}"
      ]
     },
     "execution_count": 72,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "gsearch2_2.cv_results_"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 73,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Best: -0.585230 using {'max_depth': 4, 'min_child_weight': 1}\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "D:\\anaconda\\lib\\site-packages\\sklearn\\utils\\deprecation.py:122: FutureWarning: You are accessing a training score ('mean_train_score'), which will not be available by default any more in 0.21. If you need training scores, please set return_train_score=True\n",
      "  warnings.warn(*warn_args, **warn_kwargs)\n",
      "D:\\anaconda\\lib\\site-packages\\sklearn\\utils\\deprecation.py:122: FutureWarning: You are accessing a training score ('std_train_score'), which will not be available by default any more in 0.21. If you need training scores, please set return_train_score=True\n",
      "  warnings.warn(*warn_args, **warn_kwargs)\n",
      "D:\\anaconda\\lib\\site-packages\\sklearn\\utils\\deprecation.py:122: FutureWarning: You are accessing a training score ('split0_train_score'), which will not be available by default any more in 0.21. If you need training scores, please set return_train_score=True\n",
      "  warnings.warn(*warn_args, **warn_kwargs)\n",
      "D:\\anaconda\\lib\\site-packages\\sklearn\\utils\\deprecation.py:122: FutureWarning: You are accessing a training score ('split1_train_score'), which will not be available by default any more in 0.21. If you need training scores, please set return_train_score=True\n",
      "  warnings.warn(*warn_args, **warn_kwargs)\n",
      "D:\\anaconda\\lib\\site-packages\\sklearn\\utils\\deprecation.py:122: FutureWarning: You are accessing a training score ('split2_train_score'), which will not be available by default any more in 0.21. If you need training scores, please set return_train_score=True\n",
      "  warnings.warn(*warn_args, **warn_kwargs)\n",
      "D:\\anaconda\\lib\\site-packages\\sklearn\\utils\\deprecation.py:122: FutureWarning: You are accessing a training score ('split3_train_score'), which will not be available by default any more in 0.21. If you need training scores, please set return_train_score=True\n",
      "  warnings.warn(*warn_args, **warn_kwargs)\n",
      "D:\\anaconda\\lib\\site-packages\\sklearn\\utils\\deprecation.py:122: FutureWarning: You are accessing a training score ('split4_train_score'), which will not be available by default any more in 0.21. If you need training scores, please set return_train_score=True\n",
      "  warnings.warn(*warn_args, **warn_kwargs)\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAZoAAAELCAYAAADgPECFAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMS4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvNQv5yAAAIABJREFUeJzt3XucVXW9//HXey4OKaZcxEA00DRRxBHmoGTgFdPyoCSZmv2g9Gfn2O940p8mpie1sh8aJXl6pJmXvHTR6KYeMq5eC21A8IKmSEQEIaKWeEGY+fz+WGtgz8zeM3sua88MvJ+Px37stdf6fr/rsxab72evy6yvIgIzM7OslHV1AGZmtn1zojEzs0w50ZiZWaacaMzMLFNONGZmliknGjMzy5QTjZmZZcqJxszMMuVEY2Zmmaro6gC6g/79+8eQIUO6Ogwzsx5l0aJFr0bEHq2Vc6IBhgwZQm1tbVeHYWbWo0j6SzHlfOrMzMwy5URjZmaZcqIxM7NMdUmikdRX0hxJL6XvfQqUq5O0JH3dlzP/OEmL0/mPSfpQOn+KpPU5dc4t1TaZmVl+XXVEMxWYFxH7A/PSz/m8ExHV6WtCzvwbgc9ERDXwE+CKnGX35NS5JZPozcysaF2VaE4B7kin7wBObWP9AN6fTu8GrOmkuMzMrJN1VaLZMyLWAqTvAwqU6yWpVtJCSbnJ6FxglqTVwGeBaTnLTpP0tKSZkvYuFICk89K2a9evX9/BzTEzs0Iy+zsaSXOBD+RZdHkbmtknItZI2heYL+mZiHgZuBD4eEQ8IekS4Dskyed+4KcRsUnSv5EcLR2br+GIuBm4GaCmpsbjWZtZj1BfH2yur2dLXbC5rp7NdcGW+no2b0nmb65Llr1Xl1smLVdX33h+fXDQwPcz6oN5L5N3mswSTUQcX2iZpHWSBkbEWkkDgVcKtLEmfV8h6SHgMEn/BA6NiCfSYvcAD6blNuRU/yFwbce3xMy2NxHBlvrI6ZCTjrihU95SH7y3JXlv2lFvzinbrH7a4W+p39ahJ517Tt36YHOBtt/Lt460fJJEgrr6zv1d/IWj9u25iaYV9wGTSU55TQZ+07RAeifa2+nRSX/gSOA64HVgN0kHRMSLwHjg+bTOwIZTcsCEhvlmlo26vJ1l41/bSWdZz3tpB5y/oy7QoddH8yTQtM7WjrtxR12w7bTDzpIEleVlVJaJyooyKsrK2KlcVJSXUVmuZFl5GRXpdK/KMip7VSTlKkRFWVlaRlvLbP1cVsZOFWVUlGnrvKSt5m3vVJ6WqyijsqyMyrTtnXLWvUtVeab7Arou0UwD7pV0DrAK+BSApBrg3yLiXGAY8ANJ9STXkqZFxLK03P8GfpEuex34fNruBZImAFuA14Appdsks20iggioi6A+na6P5NdofSTL69N59fU50xHU1+dMR6Qdb75fzrm/ihs67m2d7bZf5a101DnJoOXTL9vW0RBDZHzSeWsnWqa0c006y8qyxh11Q7n37ZR26GVlaeeap6NumN6aBHLaztdRN1lHo2SwdR2N65aXKdsd08Mosv6m9AA1NTXRXZ511qgDauis6hs6nWR5086qLu2sIhp3UNs6MvLMK9y55V1n5K4zqEvLb42hPhrFXtconubrzG2z6Trzd8rJOiNnO+oa1p9v+4J0fa3su6brjCb7sWnb9dHCNjVOLN1FRb7OtuCv4jydctm2X8w75XbUOXUK/XLOW67pL++cGJJ1JGUryoTkDrs7k7QoImpaK+eHanbAwy+u55r/Wdaoo2neyTbvIJt1Vk063B2BBGUS5dLW6TJBWZm2TUvp54bloqxs27QE5TnTZRLlaXnla6OsjKqKpm02bltpTA3zJVFetm26TKTraLzO3G0o31o2/zYpp42GWBvH3Xid2zr65h311sRQ4LRIRVmyvWZdyYmmA3pXlbNv/96NOr98HWRuZ9Wog8zpoMrVvKPJ7azKy5S/88ztrHI6YjVrs7UOsvk683XuLc1v3mHmWWc67V+qZjsOJ5oOGPXBvoz6bN+uDsPMrFvzQzXNzCxTTjRmZpYpJxozM8uUE42ZmWXKicbMzDLlRGNmZplyojEzs0w50ZiZWaacaMzMLFNONGZmliknGjMzy5QTjZmZZcqJxszMMuVEY2ZmmXKiMTOzTDnRmJlZppxozMwsU040ZmaWKScaMzPLlBONmZllyonGzMwy5URjZmaZcqIxM7NMOdGYmVmmnGjMzCxTTjRmZpYpJxozM8uUE42ZmWXKicbMzDLlRGNmZplyojEzs0x1SaKR1FfSHEkvpe99CpSrk7Qkfd2XM/9YSYslPSvpDkkV6XxJukHScklPSxpZqm0yM7P8uuqIZiowLyL2B+aln/N5JyKq09cEAEllwB3AGRExHPgLMDktfxKwf/o6D7gxw20wM7MidFWiOYUkWZC+n9qGuv2ATRHxYvp5DnBaTrt3RmIhsLukgZ0RsJmZtU9XJZo9I2ItQPo+oEC5XpJqJS2U1JCMXgUqJdWknycBe6fTewF/zam/Op1nZmZdpCKrhiXNBT6QZ9HlbWhmn4hYI2lfYL6kZyLiZUlnANdLqgJmA1saVpunjSgQ33kkp9fYZ5992hCSmZm1RWaJJiKOL7RM0jpJAyNibXpq65UCbaxJ31dIegg4DHg5Iv4AjE3bOgE4IK2ymm1HNwCDgTUF2r4ZuBmgpqYmbzIyM7OO66pTZ/ex7QL+ZOA3TQtI6pMesSCpP3AksCz9PCB9rwIuBW7Kafd/pXefHQH8o+EUnZmZdY2uSjTTgPGSXgLGp5+RVCPplrTMMKBW0lJgATAtIpalyy6R9DzwNHB/RMxP588CVgDLgR8C55dka8zMrCBF+KxRTU1N1NbWdnUYZmY9iqRFEVHTWjk/GcDMzDLlRGNmZplyojEzs0w50ZiZWaacaMzMLFNONGZmliknGjMzy5QTjZmZZcqJxszMMuVEY2ZmmXKiMTOzTDnRmJlZppxozMwsU040ZmaWKScaMzPLlBONmZllyonGzMwy5URjZmaZcqIxM7NMOdGYmVmmnGjMzCxTTjRmZpYpJxozM8uUE42ZmWWqoqsDMDNrUQREffJO0+n0c75paKFeS23kzC+6Hk3WnWU9aL6tLdVrmM5TL+phr5HwwY9k9s8HTjQds/5FeOGB9ENsmx850w3zc2e1VrZgG20om0kb7VhfJnG0tw2az+8p29KmjqVJZ9kZ9fK2QYE2WqpXKI4C9Sx7R37JiaZbe+U5mHd1iVeqnEm1c36p2qDA/FLH0ZY2Wou5VHE0KauydLGSZSrLP721bDpdVgaqaL1eozZo3Eaj6UL1yFO2tXoUKNtSPRXe1mLrtbS/mrXRnnoU/+/T4Xq04d+1ab10uryKrDnRdMSwCXD5um2fO7szalTGrLHNmzezevVq3n333a4OZceUHox1zQrrSrlievXqxeDBg6msrGxX/TYlGkl9gL0j4ul2rW17U1aevMy6wOrVq9l1110ZMmQI8o8Sy0hEsGHDBlavXs3QoUPb1Uard51JekjS+yX1BZYCt0v6TrvWZmad5t1336Vfv35OMpYpSfTr169DR87F3N68W0T8E/gkcHtEjAKOb/cazazTOMlYKXT0e1ZMoqmQNBA4HXigtcJmZma5ikk0XwN+ByyPiD9K2hd4KduwzKy7e+ONN/j+97/frrozZszg7bff7nAMtbW1XHDBBR1up8GUKVOYOXNms/lr1qxh0qRJADz00EOcfPLJeesPGTKEV199tdPiaXDuueeybNmyFssUin3lypX85Cc/KVjv85//PAMGDGD48OEdjrOQVhNNRPw8IkZExPnp5xURcVpmEZlZj9AdEk1NTQ033HBDh9tpzaBBg/J24qVyyy23cNBBB7WrbmuJZsqUKTz44IPtDa0ord51Juk64BvAO8CDwKHAlyLi7vauNL2x4B5gCLASOD0iXs9Trg54Jv24KiImpPOPBaYDOwGLgHMiYouko4HfAH9O6/wyIr7W3jjNeoqr73+OZWv+2altHjTo/Vz5rwcXXD516lRefvllqqurGT9+PAMGDODee+9l06ZNTJw4kauvvpq33nqL008/ndWrV1NXV8d//dd/sW7dOtasWcMxxxxD//79WbBgQd72e/fuzRe/+EXmzp1Lnz59+OY3v8mXv/xlVq1axYwZM5gwYQIPPfQQ06dP54EHHuCqq65i1apVrFixglWrVvGlL32pxaOdO++8k+nTpyOJESNGcNdddwHwyCOP8J3vfIe///3vXHfddUyaNImVK1dy8skn8+yzzzZqY8OGDZx55pmsX7+e0aNHE/n+wDl13XXX0atXLy644AIuvPBCli5dyvz585k3bx633347d999N7Nnz+bKK69k06ZN7Lffftx+++307t2bo48+munTp1NTU8Ott97Ktddey6BBg9h///2pqqrie9/7XsHYp06dyvPPP091dTWTJ0/mwgsvbBTXuHHjWLlyZcG4O0Mxp85OSG8GOBlYDRwAXNLB9U4F5kXE/sC89HM+70REdfpqSDJlwB3AGRExHPgLMDmnzqM5dZxkzDIybdo09ttvP5YsWcL48eN56aWXePLJJ1myZAmLFi3ikUce4cEHH2TQoEEsXbqUZ599lhNPPJELLriAQYMGsWDBgoJJBuCtt97i6KOPZtGiRey6665cccUVzJkzh1/96ld89atfzVvnhRde4He/+x1PPvkkV199NZs3b85b7rnnnuOaa65h/vz5LF26lO9+97tbl61du5bHHnuMBx54gKlTC3VNiauvvpqPfvSjPPXUU0yYMIFVq1YVLDtu3DgeffRRIDnlt3HjRjZv3sxjjz3G2LFjefXVV/nGN77B3LlzWbx4MTU1NXznO41v8F2zZg1f//rXWbhwIXPmzOGFF15otDxf7NOmTWPs2LEsWbKECy+8kDVr1vDxj3+8xe3qbMX8HU3DX+h8HPhpRLzWCXe6nAIcnU7fATwEXFpk3X7Apoh4Mf08B7gMuLWjQZn1VC0deZTC7NmzmT17NocddhgAGzdu5KWXXmLs2LFcfPHFXHrppZx88smMHTu26DZ32mknTjzxRAAOOeQQqqqqqKys5JBDDin4C/wTn/gEVVVVVFVVMWDAANatW8fgwYOblZs/fz6TJk2if//+APTt23frslNPPZWysjIOOugg1q1b16xurkceeYRf/vKXW9fdp0+fgmVHjRrFokWLePPNN6mqqmLkyJHU1tby6KOPcsMNN7Bw4UKWLVvGkUceCcB7773HmDFjGrXx5JNPctRRR22N91Of+hQvvvji1uXFxD5o0CBmzZrV4nZ1tmISzf2SXiA5dXa+pD2Ajv4p8p4RsRYgItZKGlCgXC9JtcAWYFpE/Bp4FaiUVBMRtcAkYO+cOmMkLQXWABdHxHMdjNXMWhERXHbZZXzhC19otmzRokXMmjWLyy67jBNOOKHg0UhTlZWVW2+rLSsro6qqauv0li1b8tZpKANQXl5esFxEFLxlN7eNlk6FNSj2h3dlZSVDhgzh9ttv5yMf+QgjRoxgwYIFvPzyywwbNoyXX36Z8ePH89Of/rRgG63F09bYS6WYmwGmAmOAmojYDLxFckTSIklzJT2b59Vq3Rz7REQNcBYwQ9J+key9M4DrJT0JvEmSiAAWAx+MiEOB/wZ+3UJ850mqlVS7fv36NoRkZgC77rorb775JgAf+9jHuO2229i4cSMAf/vb33jllVdYs2YNO++8M2effTYXX3wxixcvbla3Kxx33HHce++9bNiwAYDXXnutXe2MGzeOH//4xwD89re/5fXXm11qblZ++vTpjBs3jrFjx3LTTTdRXV2NJI444ggef/xxli9fDsDbb7/d6GgFYPTo0Tz88MO8/vrrbNmyhV/84hetxtjV+xqKezJAJfBZ4B5JM4FzgA2t1YuI4yNieJ7Xb4B16d/mkL6/UqCNNen7CpLTa4eln/8QEWMjYjTwCOnt1hHxz4jYmE7PIjny6V+g7ZsjoiYiavbYY4/WNsfMmujXrx9HHnkkw4cPZ86cOZx11lmMGTOGQw45hEmTJvHmm2/yzDPPMHr0aKqrq7nmmmu44oorADjvvPM46aSTOOaYY7ok9oMPPpjLL7+co446ikMPPZSLLrqoXe1ceeWVPPLII4wcOZLZs2ezzz77tFh+7NixrF27ljFjxrDnnnvSq1evracT99hjD370ox9x5plnMmLECI444ohm12D22msvvvKVr3D44Ydz/PHHc9BBB7Hbbru1uM4RI0ZQUVHBoYceyvXXX9/sGs2ZZ57JmDFj+NOf/sTgwYO59dbOvwqh1g6vJN1Ccp3mjnTWZ4G6iDi33SuVvgVsiIhpkqYCfSPiy03K9AHejohNabL4A3BKRCyTNCAiXpFUBcwCromI+ZI+AKyLiJA0GphJcoTT4kbW1NREbW1tezfHrEs8//zzDBs2rKvDsBLbuHEjvXv3ZsuWLUycOJHPf/7zTJw4MfP15vu+SVqUnnVqUTHXaP4lPRXVYH56DaQjpgH3SjoHWAV8CkBSDfBvaRIbBvxAUj3Jkde0iGj4i6VLJJ2czr8xIuan8ycB/y5pC8k1pTNaSzJmZj3JVVddxdy5c3n33Xc54YQTOPXUU7s6pFYVc0SzGPhURLycft4XmBkRI0sQX0n4iMZ6ou3liObwww9n06ZNjebdddddHHLIIR1ue8OGDRx33HHN5s+bN49+/fp1uP3uss5SyPqI5hJggaQVJAOnfBD4XHsCNTNr6oknnsis7X79+rFkyZLM2u8u6+zuWk00ETFP0v7Ah0kSzQtAddaBmZnZ9qGogc8iYhOwdbAzST8HWr69wszMjOIeQZOPB8EwM7OitDfR+E4uMzMrSsFEI+l+Sffled1P8rwxM9uBdYdhAjwezTbtGY/mr3/9K8cccwzDhg3j4IMPbvRw0c7U0jWa6e1cZmY7gIZEc/7557e57owZMzj77LPZeeedOxRDTU0NNTWt3l3bYd1hPJr2akg0Z511VrNlFRUVfPvb32bkyJG8+eabjBo1ivHjx7d77JtCCiaaiHi4U9dkZtn57VT4+zOtl2uLDxwCJ00ruNjj0fT88WgGDhzIwIEDgeSZaMOGDeNvf/tbpyea9l6jMbMdnMej2b7Go1m5ciVPPfUUhx9+eIvb3B5F3d5sZt1cC0cepeDxaHr2eDQbN27ktNNOY8aMGbz//e9vcZvbw4nGzDrM49G0rruOR7N582ZOO+00PvOZz/DJT36yqDptVcwwAfnuPrtL0n9K6pVJVGbW7Xk8mp4/Hk1EcM455zBs2LB2D5VQjGKu0awANgI/TF//BNYBB6SfzWwH5PFoev54NI8//jh33XUX8+fPp7q6murq6kyGeS7m6c2PRMS4fPMkPRcRXTtYeSfw05utJ9pent5sbdMTx6Mp5ohmD0lb03Q63TBq5XttCdTMzDrmqquuorq6muHDhzN06NAeMR5NMTcD/F/gMUkvkzzjbChwvqRd2DbqpplZu3g8mraZPr3n/b18McMEzEqHCTiQdJiAiHg3XTwjy+DMbPvn8Wi2f60mGkmVwBeAhus0D0n6QUTk/0soMzOzHMWcOrsRqAQanp732XTeuVkFZWZm249iEs2/RMShOZ/nS1qaVUBmZrZ9KeauszpJ+zV8kLQvUJddSGZmtj0pJtFcAiyQ9JCkh4H5JHeimdkOzOPRNNfTxqN59913GT16NIceeigHH3wwV155ZafE21SriSYi5gH7Axekrw8DHf+GmFmP1h0STU1NDTfccEOH22lNdxiPpr2P7m8p0VRVVW19gvWSJUt48MEHWbhwYUdCzauoh2pGxCbg6YbPkn4OtPysBTMrmWufvJYXXnuh9YJtcGDfA7l09KUFl3s8mp4/Ho0kevfuDSQP19y8eXPRDwlti/aOR9P5kZhZj+LxaLaP8Wjq6uqorq5mwIABjB8/vluNR1Pc86fNrCRaOvIoBY9H03PHoykvL2fJkiW88cYbTJw4kWeffZbhw4e3uN1tVTDRSLqf/AlFQDbPbjCzHsnj0bSuu45H02D33Xfn6KOP5sEHH+z0RNPSqbPpwLfzvKYDzccBNbMdisej6fnj0axfv5433ngDgHfeeYe5c+dy4IEHttpmWxU8oomIhzt9bWa23cgdj+akk07aOh4NJBfy7777bpYvX84ll1xCWVkZlZWV3HjjjcC28WgGDhzY4nWarOSOR1NeXs5hhx3Gj370oza3c+WVV3LmmWcycuRIjjrqqKLGo7nmmmsYM2YMu+yyS8HxaBoeMvqNb3yDAw44YGv93PFoBg0a1ObxaKZMmcKnP/1pzj33XGbNmsXatWuZPHkydXV11NfXc/rppxe8dbsjWh2PZkfg8WisJ/J4NDum7XU8GjMz6ya21/FotpL0gYj4e1bBmNmOx+PRtM12OR5NE7OAkVkEYmY7Jo9Hs/1r66kz/6GmWTfia6xWCh39nrU10fywQ2szs07Tq1cvNmzY4GRjmYoINmzYQK9evdrdRptOnUVE+56gl4ekvsA9wBBgJXB6RDS7CV3SPsAtwN4kf0D68YhYKWko8DOgL7AY+GxEvCepCrgTGAVsAD4dESs7K26z7mLw4MGsXr2a9evXd3Uotp3r1atX3icsFKu9j6DpDFOBeRExTdLU9HO+52jcCVwTEXMk9Qbq0/nXAtdHxM8k3QScQzLy5znA6xHxIUlnpOU+nfXGmJVaZWUlQ4cO7eowzFrVlbc3nwLckU7fATS7R0/SQUBFRMwBiIiNEfG2kmc+HAvMzFM/t92ZwHHK4nGkZmZWlK5MNHtGxFqA9H1AnjIHAG9I+qWkpyR9S1I5ybPW3oiIhgcZrQb2Sqf3Av6atrsF+Ad+NpuZWZfJ9NSZpLnAB/IsurzIJiqAscBhwCqSazpTgPvylG24Iprv6KXZ1VJJ5wHnAa0+NsLMzNov00QTEccXWiZpnaSBEbFW0kDglTzFVgNPRcSKtM6vgSOA24DdJVWkRy2DgTU5dfYGVkuqAHYDmj0xLyJuBm6G5BE07d1GMzNrWVeeOrsPmJxOTwZ+k6fMH4E+kvZIPx8LLIvkfs4FwKQ89XPbnQTMD9//aWbWZboy0UwDxkt6CRiffkZSjaRbACKiDrgYmCfpGZLTYg1/y3MpcJGk5STXYG5N598K9EvnX0RyN5uZmXURP70ZP73ZzKw9/PRmMzPrFpxozMwsU040ZmaWKScaMzPLlBONmZllyonGzMwy5URjZmaZcqIxM7NMOdGYmVmmnGjMzCxTTjRmZpYpJxozM8uUE42ZmWXKicbMzDLlRGNmZplyojEzs0w50ZiZWaacaMzMLFNONGZmliknGjMzy5QTjZmZZcqJxszMMuVEY2ZmmXKiMTOzTDnRmJlZppxozMwsU040ZmaWKScaMzPLlBONmZllyonGzMwy5URjZmaZcqIxM7NMOdGYmVmmnGjMzCxTXZJoJPWVNEfSS+l7nwLl9pE0W9LzkpZJGpLOHyrpibT+PZJ2SudPkbRe0pL0dW7ptsrMzPLpqiOaqcC8iNgfmJd+zudO4FsRMQwYDbySzr8WuD6t/zpwTk6deyKiOn3dkk34ZmZWrK5KNKcAd6TTdwCnNi0g6SCgIiLmAETExoh4W5KAY4GZLdU3M7PuoasSzZ4RsRYgfR+Qp8wBwBuSfinpKUnfklQO9APeiIgtabnVwF459U6T9LSkmZL2znIjzMysdZklGklzJT2b53VKkU1UAGOBi4F/AfYFpgDKUzbS9/uBIRExApjLtqOmfPGdJ6lWUu369euLDMnMzNqqIquGI+L4QsskrZM0MCLWShrItmsvuVYDT0XEirTOr4EjgNuA3SVVpEc1g4E16To35NT/Icm1nELx3QzcDFBTUxOFypmZWcd01amz+4DJ6fRk4Dd5yvwR6CNpj/TzscCyiAhgATCpaf00aTWYADzfyXGbmVkbdVWimQaMl/QSMD79jKQaSbcAREQdyWmzeZKeITll9sO0/qXARZKWk1yzuTWdf4Gk5yQtBS4gOdVmZmZdSMkBwo6tpqYmamtruzoMM7MeRdKiiKhprZyfDGBmZplyojEzs0w50ZiZWaacaMzMLFNONGZmliknGjMzy5QTjZmZZcqJxszMMuVEY2ZmmXKiMTOzTDnRmJlZppxozMwsU040ZmaWKScaMzPLlBONmZllyonGzMwy5URjZmaZcqIxM7NMOdGYmVmmnGjMzCxTFV0dQE/2hzV/4LuLv9timSBaXh4tLy9Ga+soZj1FtVGKbSmijVbj6IT9UYzO2B/FxJqPUPN5yjMvT7m87RVZt9j1Zt1+V21X3vXmndV63ZJsVxHlThp6EpMOmFTUetvLiaYDdirfib69+rZarrX/iMV8uVotU8T3s7U2ioqjM7aliI6p1TZa25ZiOr9O2B+t/7N0/v7Il8DyJqy8s4qrW/Q6ioivI+131XZ1aL1FrKNU29Xs3yJPufqob77iTuZE0wGj9hzFqD1HdXUYZmbdmq/RmJlZppxozMwsU040ZmaWKScaMzPLlBONmZllyonGzMwy5URjZmaZcqIxM7NMqTMexdHTSVoP/KWd1fsDr3ZiOJ2lu8YF3Tc2x9U2jqtttse4PhgRe7RWyImmgyTVRkRNV8fRVHeNC7pvbI6rbRxX2+zIcfnUmZmZZcqJxszMMuVE03E3d3UABXTXuKD7xua42sZxtc0OG5ev0ZiZWaZ8RGNmZplyoilA0t6SFkh6XtJzkv4zTxlJukHScklPSxqZs2yypJfS1+QSx/WZNJ6nJf1e0qE5y1ZKekbSEkm1JY7raEn/SNe9RNJXc5adKOlP6b6cWuK4LsmJ6VlJdZL6psuy2l+9JD0paWka19V5ylRJuifdJ09IGpKz7LJ0/p8kfazEcV0kaVn6/Zon6YM5y+py9uV9JY5riqT1Oes/N2dZVv8fi4nr+pyYXpT0Rs6yTPZXTvvlkp6S9ECeZaX7fkWEX3lewEBgZDq9K/AicFCTMh8HfksyzuIRwBPp/L7AivS9Tzrdp4RxfaRhfcBJDXGln1cC/btofx0NPJCnbjnwMrAvsBOwtGndLONqUv5fgfkl2F84h8GVAAAG6ElEQVQCeqfTlcATwBFNypwP3JROnwHck04flO6jKmBouu/KSxjXMcDO6fS/N8SVft7Y2fuqDXFNAb6Xp26W/x9bjatJ+f8Abst6f+W0fxHwkwL/70r2/fIRTQERsTYiFqfTbwLPA3s1KXYKcGckFgK7SxoIfAyYExGvRcTrwBzgxFLFFRG/T9cLsBAY3Bnr7mhcLRgNLI+IFRHxHvAzkn3bFXGdCfy0M9bdSlwRERvTj5Xpq+kF01OAO9LpmcBxkpTO/1lEbIqIPwPLSfZhSeKKiAUR8Xb6sVTfr2L2VyFZ/n9sa1wl+X4BSBoMfAK4pUCRkn2/nGiKkB5SHkbyayXXXsBfcz6vTucVml+quHKdQ3LU1SCA2ZIWSTqvs2MqIq4x6WmG30o6OJ3XLfaXpJ1JOqBf5MzObH+lpzWWAK+QdIQFv18RsQX4B9CPjPdXEXHlavr96iWpVtJCSad2VkxtiOu09JTeTEl7p/O6xf5KTzEOBebnzM5sfwEzgC8D9QWWl+z75UTTCkm9STqeL0XEP5suzlMlWphfqrgayhxD0hFcmjP7yIgYSXJK7YuSxpUwrsUkj6w4FPhv4NcN1fI0VfL9RXLa7PGIeC1nXmb7KyLqIqKa5IhgtKThTcPOV62F+aWKKwlOOhuoAb6VM3ufSP7K/CxghqT9ShjX/cCQiBgBzGXbr/Vusb9ITk/NjIi6nHmZ7C9JJwOvRMSilorlmZfJ98uJpgWSKkk6px9HxC/zFFkN7J3zeTCwpoX5pYoLSSNIDplPiYgNDfMjYk36/grwKzrplEsxcUXEPxtOM0TELKBSUn+6wf5KnUGT0xpZ7q+cdbwBPETz0zlb94ukCmA34DUy3l9FxIWk44HLgQkRsSmnTsP+WpHWPaxUcUXEhpxYfgiMSqe7fH+lWvp+dfb+OhKYIGklyanoYyXd3aRM6b5fHbnAsz2/SLL6ncCMFsp8gsY3AzyZzu8L/JnkwmOfdLpvCePah+S86keazN8F2DVn+vfAiSWM6wNs+9ut0cCqtF4FyQXaoWy7GeDgUsWVlmv4T7ZLifbXHsDu6fT7gEeBk5uU+SKNL9bem04fTOOLtSvovJsBionrMJILxPs3md8HqEqn+wMv0Xk3dRQT18Cc6YnAwnQ6y/+PrcaVLvswyY0lKsX+arLuo8l/M0DJvl8VWCFHAp8FnknPvwJ8haQTJyJuAmaR3Hm2HHgb+Fy67DVJXwf+mNb7WjQ+HZN1XF8lOdf6/eTaHlsiOTzfE/hVOq8C+ElEPFjCuCYB/y5pC/AOcEYk3+wtkv4P8DuSO9Bui4jnShgXJB3T7Ih4K6dulvtrIHCHpHKSMwv3RsQDkr4G1EbEfcCtwF2SlpMkwTPSmJ+TdC+wDNgCfDEan47JOq5vAb2Bn6f7ZlVETACGAT+QVJ/WnRYRy0oY1wWSJpDsk9dI7kLL+v9jMXFBchPAz9Lve4Ms91deXfX98pMBzMwsU75GY2ZmmXKiMTOzTDnRmJlZppxozMwsU040ZmaWKScaMzPLlBONWQ+iZNiC/u2sO0XSoM5oy6wtnGjMdhxTgEGtFTLrbE40Zu0gaYikFyTdomSwtB9LOl7S40oG1xqdvn6fDjz1e0kfTuteJOm2dPqQtP7OBdbTT9LstI0fkPPAQ0lnKxl0a4mkH6R/nY6kjZK+LWmxkoHJ9pA0ieQBmD9Oy78vbeY/0nLPSDowy31mOy4nGrP2+xDwXWAEcCDJE3g/ClxM8pibF4BxEXEYyWOBvpnWmwF8SNJE4HbgC7FtfJemrgQeS9u4j/TROZKGAZ8mebp0NVAHfCatswuwOJKnTj8MXBkRM4Fa4DMRUR0R76RlX03L3ZjGbdbp/Kwzs/b7c0Q8AyDpOWBeRISkZ4AhJA/qvEPS/iSPWa8EiIh6SVOAp4EfRMTjLaxjHPDJtN7/SGoY0O44kqcT/zF93tj7SMZDgWT8kXvS6buBlp5Y3bBsUcN6zDqbE41Z+23Kma7P+VxP8n/r68CCiJioZNC1h3LK7w9spLhrJvkeSCjgjoi4rJ31GzTEXIf7A8uIT52ZZWc34G/p9JSGmZJ2IznlNg7ol14/KeQR0lNikk4iebQ8wDxgkqQB6bK+6QiOkPy/bmjzLOCxdPpNYNcObI9ZuzjRmGXnOuD/SXqcZPiDBtcD34+IF0lGQJ3WkDDyuBoYJ2kxcALJGD6kj5O/gmSY6aeBOSSPrAd4CzhY0iLgWOBr6fwfATc1uRnALHMeJsBsOyNpY0T07uo4zBr4iMbMzDLlIxqzbkDS54D/bDL78Yj4YlfEY9aZnGjMzCxTPnVmZmaZcqIxM7NMOdGYmVmmnGjMzCxTTjRmZpap/w/Dtf68tJdNkQAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x16590d16518>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# summarize results\n",
    "print(\"Best: %f using %s\" % (gsearch2_2.best_score_, gsearch2_2.best_params_))\n",
    "test_means = gsearch2_2.cv_results_[ 'mean_test_score' ]\n",
    "test_stds = gsearch2_2.cv_results_[ 'std_test_score' ]\n",
    "train_means = gsearch2_2.cv_results_[ 'mean_train_score' ]\n",
    "train_stds = gsearch2_2.cv_results_[ 'std_train_score' ]\n",
    "\n",
    "pd.DataFrame(gsearch2_2.cv_results_).to_csv('my_preds_maxdepth_min_child_weights_2.csv')\n",
    "\n",
    "# plot results\n",
    "test_scores = np.array(test_means).reshape(len(min_child_weight), len(max_depth))\n",
    "train_scores = np.array(train_means).reshape(len(min_child_weight), len(max_depth))\n",
    "\n",
    "for i, value in enumerate(min_child_weight):\n",
    "    pyplot.plot(max_depth, test_scores[i], label= 'test_min_child_weight:'   + str(value))\n",
    "#for i, value in enumerate(min_child_weight):\n",
    "#    pyplot.plot(max_depth, train_scores[i], label= 'train_min_child_weight:'   + str(value))\n",
    "    \n",
    "pyplot.legend()\n",
    "pyplot.xlabel( 'max_depth' )                                                                                                      \n",
    "pyplot.ylabel( '- Log Loss' )\n",
    "pyplot.savefig( 'max_depth_vs_min_child_weght2.png' )"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "max_depth': 4, 'min_child_weight': 1  效果还没有之前好，差距影响不大。后续采取4，当然也可使用5，效果都测试过，差距不大。\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 重新调整分类器数目"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 74,
   "metadata": {},
   "outputs": [],
   "source": [
    "#再次调整弱分类器数目\n",
    "def modelfit(alg, X_train, y_train, useTrainCV=True, cv_folds=None, early_stopping_rounds=100):\n",
    "    \n",
    "    if useTrainCV:\n",
    "        xgb_param = alg.get_xgb_params()\n",
    "        xgb_param['num_class'] = 3\n",
    "        \n",
    "        xgtrain = xgb.DMatrix(X_train, label = y_train)\n",
    "        \n",
    "        cvresult = xgb.cv(xgb_param, xgtrain, num_boost_round=alg.get_params()['n_estimators'], folds =cv_folds,\n",
    "                         metrics='mlogloss', early_stopping_rounds=early_stopping_rounds)\n",
    "        \n",
    "        n_estimators = cvresult.shape[0]\n",
    "        alg.set_params(n_estimators = n_estimators)\n",
    "        \n",
    "        print (cvresult)\n",
    "        #result = pd.DataFrame(cvresult)   #cv缺省返回结果为DataFrame\n",
    "        #result.to_csv('my_preds.csv', index_label = 'n_estimators')\n",
    "        cvresult.to_csv('my_preds4_2_3_699.csv', index_label = 'n_estimators')\n",
    "        \n",
    "        # plot\n",
    "        test_means = cvresult['test-mlogloss-mean']\n",
    "        test_stds = cvresult['test-mlogloss-std'] \n",
    "        \n",
    "        train_means = cvresult['train-mlogloss-mean']\n",
    "        train_stds = cvresult['train-mlogloss-std'] \n",
    "\n",
    "        x_axis = range(0, n_estimators)\n",
    "        pyplot.errorbar(x_axis, test_means, yerr=test_stds ,label='Test')\n",
    "        pyplot.errorbar(x_axis, train_means, yerr=train_stds ,label='Train')\n",
    "        pyplot.title(\"XGBoost n_estimators vs Log Loss\")\n",
    "        pyplot.xlabel( 'n_estimators' )\n",
    "        pyplot.ylabel( 'Log Loss' )\n",
    "        pyplot.savefig( 'n_estimators4_2_3_699.png' )\n",
    "    \n",
    "    #Fit the algorithm on the data\n",
    "    alg.fit(X_train, y_train, eval_metric='mlogloss')\n",
    "        \n",
    "    #Predict training set:\n",
    "    train_predprob = alg.predict_proba(X_train)\n",
    "    logloss = log_loss(y_train, train_predprob)\n",
    "\n",
    "        \n",
    "    #Print model report:\n",
    "    print(\"logloss of train :\" )\n",
    "    print(logloss)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 75,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "     test-mlogloss-mean  test-mlogloss-std  train-mlogloss-mean  \\\n",
      "0              1.041480           0.000141             1.041126   \n",
      "1              0.994425           0.001048             0.993702   \n",
      "2              0.952905           0.001114             0.952009   \n",
      "3              0.917690           0.000832             0.916356   \n",
      "4              0.886846           0.000789             0.885102   \n",
      "5              0.859715           0.000611             0.857773   \n",
      "6              0.836183           0.000660             0.833921   \n",
      "7              0.815741           0.000707             0.813102   \n",
      "8              0.797870           0.001015             0.795012   \n",
      "9              0.781764           0.000950             0.778553   \n",
      "10             0.767685           0.000935             0.764180   \n",
      "11             0.755084           0.000790             0.751346   \n",
      "12             0.743946           0.000950             0.739978   \n",
      "13             0.734065           0.001032             0.729795   \n",
      "14             0.725044           0.000922             0.720474   \n",
      "15             0.717200           0.000868             0.712289   \n",
      "16             0.709905           0.000837             0.704731   \n",
      "17             0.703647           0.001072             0.698160   \n",
      "18             0.697735           0.001180             0.691977   \n",
      "19             0.692480           0.001371             0.686428   \n",
      "20             0.687642           0.001489             0.681375   \n",
      "21             0.683256           0.001387             0.676746   \n",
      "22             0.679430           0.001603             0.672596   \n",
      "23             0.675928           0.001728             0.668768   \n",
      "24             0.672530           0.001657             0.665028   \n",
      "25             0.669620           0.001765             0.661771   \n",
      "26             0.666776           0.001797             0.658656   \n",
      "27             0.664239           0.001920             0.655871   \n",
      "28             0.661907           0.001817             0.653194   \n",
      "29             0.659570           0.001759             0.650574   \n",
      "..                  ...                ...                  ...   \n",
      "310            0.588158           0.002014             0.520132   \n",
      "311            0.588091           0.002051             0.519907   \n",
      "312            0.588055           0.002069             0.519710   \n",
      "313            0.588035           0.002072             0.519473   \n",
      "314            0.587988           0.002074             0.519196   \n",
      "315            0.588006           0.002076             0.519015   \n",
      "316            0.587981           0.002074             0.518792   \n",
      "317            0.587937           0.002059             0.518523   \n",
      "318            0.587900           0.002049             0.518280   \n",
      "319            0.587877           0.002029             0.518086   \n",
      "320            0.587799           0.002008             0.517847   \n",
      "321            0.587768           0.001994             0.517664   \n",
      "322            0.587760           0.001999             0.517500   \n",
      "323            0.587747           0.002012             0.517259   \n",
      "324            0.587708           0.002015             0.517058   \n",
      "325            0.587649           0.002023             0.516824   \n",
      "326            0.587633           0.002012             0.516631   \n",
      "327            0.587593           0.001995             0.516447   \n",
      "328            0.587518           0.002019             0.516202   \n",
      "329            0.587499           0.002008             0.515986   \n",
      "330            0.587468           0.001992             0.515778   \n",
      "331            0.587491           0.001984             0.515565   \n",
      "332            0.587446           0.001992             0.515389   \n",
      "333            0.587428           0.002013             0.515245   \n",
      "334            0.587401           0.002026             0.515054   \n",
      "335            0.587356           0.002054             0.514875   \n",
      "336            0.587269           0.002045             0.514648   \n",
      "337            0.587236           0.002047             0.514433   \n",
      "338            0.587225           0.002072             0.514248   \n",
      "339            0.587156           0.002093             0.514025   \n",
      "\n",
      "     train-mlogloss-std  \n",
      "0              0.000199  \n",
      "1              0.001177  \n",
      "2              0.001435  \n",
      "3              0.001387  \n",
      "4              0.001390  \n",
      "5              0.001137  \n",
      "6              0.000938  \n",
      "7              0.001138  \n",
      "8              0.001339  \n",
      "9              0.001196  \n",
      "10             0.001111  \n",
      "11             0.001199  \n",
      "12             0.000933  \n",
      "13             0.000951  \n",
      "14             0.001112  \n",
      "15             0.001138  \n",
      "16             0.001264  \n",
      "17             0.001080  \n",
      "18             0.001084  \n",
      "19             0.000942  \n",
      "20             0.000835  \n",
      "21             0.000959  \n",
      "22             0.000799  \n",
      "23             0.000740  \n",
      "24             0.000909  \n",
      "25             0.000933  \n",
      "26             0.000988  \n",
      "27             0.000988  \n",
      "28             0.001095  \n",
      "29             0.001129  \n",
      "..                  ...  \n",
      "310            0.000986  \n",
      "311            0.001009  \n",
      "312            0.001015  \n",
      "313            0.001021  \n",
      "314            0.001012  \n",
      "315            0.001037  \n",
      "316            0.000988  \n",
      "317            0.001007  \n",
      "318            0.000996  \n",
      "319            0.001032  \n",
      "320            0.001087  \n",
      "321            0.001049  \n",
      "322            0.001031  \n",
      "323            0.001022  \n",
      "324            0.000996  \n",
      "325            0.000989  \n",
      "326            0.000974  \n",
      "327            0.000951  \n",
      "328            0.000915  \n",
      "329            0.000975  \n",
      "330            0.000956  \n",
      "331            0.000941  \n",
      "332            0.000874  \n",
      "333            0.000888  \n",
      "334            0.000900  \n",
      "335            0.000879  \n",
      "336            0.000865  \n",
      "337            0.000869  \n",
      "338            0.000845  \n",
      "339            0.000878  \n",
      "\n",
      "[340 rows x 4 columns]\n",
      "logloss of train :\n",
      "0.531658267215\n"
     ]
    },
    {
     "ename": "TypeError",
     "evalue": "cross_val_score() got an unexpected keyword argument 'metrics'",
     "output_type": "error",
     "traceback": [
      "\u001b[1;31m---------------------------------------------------------------------------\u001b[0m",
      "\u001b[1;31mTypeError\u001b[0m                                 Traceback (most recent call last)",
      "\u001b[1;32m<ipython-input-75-003b90ab8f0a>\u001b[0m in \u001b[0;36m<module>\u001b[1;34m()\u001b[0m\n\u001b[0;32m     14\u001b[0m \u001b[0mmodelfit\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0mxgb2_3\u001b[0m\u001b[1;33m,\u001b[0m \u001b[0mX_train\u001b[0m\u001b[1;33m,\u001b[0m \u001b[0my_train\u001b[0m\u001b[1;33m,\u001b[0m \u001b[0mcv_folds\u001b[0m \u001b[1;33m=\u001b[0m \u001b[0mkfold\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m     15\u001b[0m \u001b[1;32mfrom\u001b[0m \u001b[0msklearn\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mmodel_selection\u001b[0m \u001b[1;32mimport\u001b[0m \u001b[0mcross_val_score\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[1;32m---> 16\u001b[1;33m \u001b[0mresults\u001b[0m \u001b[1;33m=\u001b[0m \u001b[0mcross_val_score\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0mxgb2_3\u001b[0m\u001b[1;33m,\u001b[0m \u001b[0mX_train\u001b[0m\u001b[1;33m,\u001b[0m \u001b[0my_train\u001b[0m\u001b[1;33m,\u001b[0m \u001b[0mmetrics\u001b[0m\u001b[1;33m=\u001b[0m\u001b[1;34m'mlogloss'\u001b[0m\u001b[1;33m,\u001b[0m \u001b[0mcv\u001b[0m\u001b[1;33m=\u001b[0m\u001b[0mkfold\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0m\u001b[0;32m     17\u001b[0m \u001b[0mprint\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0mresults\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m     18\u001b[0m \u001b[0mprint\u001b[0m\u001b[1;33m(\u001b[0m\u001b[1;34m\"CV logloss: %.2f%% (%.2f%%)\"\u001b[0m \u001b[1;33m%\u001b[0m \u001b[1;33m(\u001b[0m\u001b[0mresults\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mmean\u001b[0m\u001b[1;33m(\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m*\u001b[0m\u001b[1;36m100\u001b[0m\u001b[1;33m,\u001b[0m \u001b[0mresults\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mstd\u001b[0m\u001b[1;33m(\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m*\u001b[0m\u001b[1;36m100\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n",
      "\u001b[1;31mTypeError\u001b[0m: cross_val_score() got an unexpected keyword argument 'metrics'"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYYAAAEXCAYAAACpuuMDAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMS4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvNQv5yAAAIABJREFUeJzt3XmcHHWd//HXp3vuK5lkJpADknCFaznDJSgIHoAKgoAguODFT3/itev6Q9dVFmVV1nW9WF1WgfUCEVERUZRbEZBwnwkh5L4m51yZq+fz++NbnamZdM/0JNPTPen38/GodHdVdfWnqyf97vpW1bfM3REREUlLFLoAEREpLgoGEREZRMEgIiKDKBhERGQQBYOIiAyiYBARkUEUDCIxZvY5M/tBoesQKSQFwwRjZnVmttTM3hMbV29my83svNi4+WZ2p5ltNrMtZvaimV1jZo3R9MvMLGVm7dGwxMw+kufaTzGzlfl8jdHIVI+7/5u7fzBPr7fUzN6Uj2Xnw3h9XhNtvZQCBcME4+7twOXAt8ysORp9LbDA3W8DMLPXAQ8ADwMHuvtk4HSgDzg8trhH3L3O3euA84BrzezI8XknMhpmVlboGqSEuLuGCTgANwE3A6cAG4HpsWl/Ab4zwvMvA/4yZNzfgPfEHp8FvABsIQTNQbFpB0XjtkTznBWbdibwItAGrAI+DdQC24B+oD0aZmR5X9cBv4ue/xiwbw7r40DgT8AmYCFwwc7UA1wF/CR63hzAgfcBK4DNwIeBY4Bno/f+3djr7AvcF30eG4CfApOjaT+OXmtb9FqfyWEdLwX+X/Ra3UBZ9HhV9F4WAqdlWBfHA2uBZGzcOcCz0f1jgQVAK7AO+EaWdXoKsDLLtEnAj4AWYBnweSARTUsC/xGtg9eAK6L1WJZlWUuBN2WZ9iFgcfS53pH+mwEM+E9gPbA1WkeHZvu8C/3/daINBS9Aw05+cNAIrIn+870vNr4WSAGnjPD8y4gFQ/RltwU4IHp8ANABvBkoBz4T/QetiB4vBj4XPT41+k84L3ruGuD1sTqPiu5n/aKJ1XFT9CVwbPRF+FPglhGeU0v44n5f9JyjovVyyGjrIXMwfB+oAt4CdAG/BqYBM6MvppOj+feL1lcl0Aw8BHwztuxBX4DDrePY/E8DewHVwLzofc6I1ZcxNIFXgTfHHv8CuDK6/wjw3uh+HXB8lmVk/bwIofAboD6qYxHwgWjahwlfzLOi9X0POxEM0d/VhujzrAS+AzwUTXsr8AQwmRASBxH9OMr2eWvIfVBT0gTl7psJvzRrgNtjkxoJTYRr0yPM7NpoP0OHmX0+Nu/x0fh2wtbCj4FXomnvBn7n7n9y917g64Qvp9cRfpHWAV919x53vw+4E7goem4vcLCZNbj7Znd/cpRv73Z3/5u79xGC4YgR5n87sNTdb3T3vuj1fkloHhuLer7k7l3u/kfCF/nN7r7e3VcBfwaOBHD3xdH66nb3FuAbwMnDLHe4dZz2bXdf4e7bCIFfGb2Xcndf6u6vZln2zUSfh5nVE35F3xxbH/uZWZO7t7v7o6NZGWaWjGr/rLu3uftSwhbCe6NZLgC+5e4ro7/Tr45m+TEXAze4+5Pu3g18FjjBzOZE76GesKVo7v6Su6+Jvb9d+bxLnoJhgjKzSwi/1O4BvhabtJnQZDE9PcLdP+NhP8OvCL+o0x5198ke9jHsCRwC/Fs0bQahiSC9jH7Cr9WZ0bQV0bi0ZdE0gHcRvoiWmdmDZnbCKN/e2tj9TkIIDWc2cFwUclvMbAvhS2XPMapnXez+tgyP6wDMbJqZ3WJmq8ysFfgJ0DTMcodbx2krYtMXA58kbNWsj15rRpZl/ww418wqgXOBJ909/VofIGytvGxmj5vZ24epMZMmwpbisti4+Oc/I173kPujMXT9tBOa6WZGP0a+S2h2XGdm15tZQzTrrn7eJU/BMAGZ2TRC++qHgP8DXGBmbwBw9w5Cu/y5o1mmu68j/Mp+RzRqNeELN/2aRmjSWBVN28vM4n8/e0fTcPfH3f1sQnPLr4Fb0y8zmppGYQXwYBRy6aHO3T8yzvV8JVrmYe7eAFxCaOZIG/p6w63jjM9x95+5+0nR85zBPwri871I+FI9A3gPISjS015x94sI6+NrwG1mVpv722QD4Vf57Ni47Z8/oSlnVmzaXqNYdtzQ9VMLTGXg7+zb7n404QfNAcA/ReOzfd6SIwXDxPRd4Nfufn+0+fwZ4H+iX4dEj99vZldGIYKZzQLmZlugmU0l7KB8IRp1K/A2MzvNzMqBfyTsAP0rIXg6gM+YWbmZnUIIlFvMrMLMLjazSVHzSCuhCQTCL+2pZjZpjNZD2p3AAWb23qiecjM7xswOGud66gk7lreY2UyiL6qYdcA+scfDreMdmNk8Mzs1+py7CFsrqUzzRn4GfBx4A2EfQ3o5l5hZc7SFsiUanXU5ZlYVHwhbpLcC10SHSs8G/oGwhZR+X58ws5lmNpmww3wk5UNepyyq/31mdkT0nv8NeMzdl0af73HReuuI1kdqhM9bclXonRwaRjcA7yT8kpo8ZPy9wDWxx8cBdxH+428BngeuAaZG0y8j/IdJH5GzntAGPS22jHMIOxG3Ag8S7cyNph0SjdsazXNONL4C+AOhSasVeBw4Kfa8GwjNAVvIflTSl2OPT2GEHdbRfPMIRzK1RMu/j7BvYlT1kHnnc1ls/pXEduwTvgw/H1snT0Tr82nCF/3K2LxnA8uj1/p0Dut4KYN3Vh9G2BfURthBf2emdRibf2/Cl/jvhoz/SfR5txN+CLwzy/NPid7/0GE/wr6sn0TrewXwBQaOSiojbNFuJByV9CnCFoZleZ2lGV7jy9G0DxN2pKff76xo/GmEI5HaGTgCrG6kz1tDboNFK1lEJC/M7Azg++4+e8SZpSioKUlExpSZVZvZmWZWFjWpfZFw4INMENpikAnBzF4P/D7TNA9HVUmRMLMaQrPYgYT9IL8DPuHurQUtTHKmYBARkUHUlCQiIoNMuI65mpqafM6cOYUuQ0RkQnniiSc2uHvzyHNOwGCYM2cOCxYsKHQZIiITipktG3muQE1JIiIyiIJBREQGUTCIiMggCgYRERlEwSAiIoMoGEREZBAFg4iIDFIywbBkySs88Mdf4/39I88sIlLCSiYYWm75OKf89VI6O9sLXYqISFErmWCw/U8DoLNtywhzioiUtpIJhmR1PQCdbZsLXImISHErmWAoq24AoKt9a4ErEREpbiUTDBU14XrvPR0KBhGR4ZRMMFTWTgagd5uCQURkOCUTDFV1IRj6OnV1QRGR4ZRMMNTUhaak/i5tMYiIDKd0gqGhEQDvaitwJSIixa1kgqGispoeT+I9OsFNRGQ4JRMMmNFpNSS6tcUgIjKc0gkGCMHQqy0GEZHhlFQwdCWqKVcwiIgMK2/BYGY3mNl6M3s+y3Qzs2+b2WIze9bMjspXLWndiVrKUp35fhkRkQktn1sMNwGnDzP9DGD/aLgc+F4eawGgt6yWylRHvl9GRGRCy1swuPtDwKZhZjkb+JEHjwKTzWx6vuoB6Curo6pfwSAiMpxC7mOYCayIPV4ZjduBmV1uZgvMbEFLS8tOv2CqopYaV1OSiMhwChkMlmGcZ5rR3a939/nuPr+5uXmnX7C/vJ4a37bTzxcRKQWFDIaVwF6xx7OA1Xl9xU1LqLFuunt68voyIiITWSGD4Q7g76Ojk44Htrr7mny+oM05CYDONvWXJCKSTVm+FmxmNwOnAE1mthL4IlAO4O7fB+4CzgQWA53A+/JVS1qiOnSk19m2mcapO98kJSKyO8tbMLj7RSNMd+Cj+Xr9TNKX9+xq13WfRUSyKakzn9NXcevWVdxERLIqyWDo6VQwiIhkU1LBUBldxa1XwSAiklVJBUNNfQiG1DZd3lNEJJuSCobqaIuhv0vBICKSTUkFQ/q6z+hiPSIiWZVUMFiyjE4qoVvXZBARyaakggGgA13FTURkOCUXDF2JGsoUDCIiWZVcMHQnaijvUzCIiGRTesGQrKW8TxfrERHJpuSCYdu2bVT0qK8kEZFsSi4YmLovdQldj0FEJJuSC4ZU5WQavI3QuauIiAxVcsFAdSM11s22bdrPICKSSckFQ6JmCgBtm1sKXImISHEquWBI1k0FoGOLgkFEJJOSC4bKurDFsK11Y4ErEREpTqUXDJPCtZ67WzcUuBIRkeJUcsFQO6kJgFS7thhERDIpuWCoawxbDP2dmwpciYhIcSq9YKibRLeXwbbNhS5FRKQolVwwWCJBq9WR6FYwiIhkUnLBANCRqKe8W/0liYhkUlboAgphQ6qW/nbtYxARyaQkgyFhxhRXU5KISCYl2ZTUN3kuddZV6DJERIpSSQZDf1Uj9d6uHlZFRDLIazCY2elmttDMFpvZlRmmzzaze83sWTN7wMxm5bOetERNI9XWQ3t723i8nIjIhJK3YDCzJHAdcAZwMHCRmR08ZLavAz9y98OAq4Gv5KueuHRHels3rR+PlxMRmVDyucVwLLDY3Ze4ew9wC3D2kHkOBu6N7t+fYXpeVNaHYGhV19siIjvIZzDMBFbEHq+MxsU9A7wrun8OUG9mU4cuyMwuN7MFZragpWXXv8yrG0J/SdvU9baIyA7yGQyWYdzQvb2fBk42s6eAk4FVQN8OT3K/3t3nu/v85ubmXS6sdvI0ALrUw6qIyA7yeR7DSmCv2ONZwOr4DO6+GjgXwMzqgHe5+9Y81gTApKkhGHraFQwiIkPlc4vhcWB/M5trZhXAhcAd8RnMrMnM0jV8Frghj/Vsl25K6u/QSW4iIkPlLRjcvQ+4ArgbeAm41d1fMLOrzeysaLZTgIVmtgjYA7gmX/UMUl5DN+VYl4JBRGSovHaJ4e53AXcNGfeF2P3bgNvyWUNGZrR7Nf0ti8b9pUVEil1J9pUE0FHRRF3pvn0RkaxKsksMgK6KKdT1qSlJRGSokg2GnqqpTOrfov6SRESGKNlgSFU3MYWtdPakCl2KiEhRKdlgsLpp1Fo3m7foSm4iInElGwxl9eEkt9aNq0eYU0SktJRsMFQ27glA5+a1Ba5ERKS4lGww1EbB0KVgEBEZpGSDoaEpdPTa16ZrMoiIxJVsMFRHPay+vPjVAlciIlJcSjYYrLyaNq/mIF9c6FJERIpKSfcJsbWsmcryikKXISJSVEp2iwGgvbKJuh5dxU1EJK6kg6GnehqNqU3qFkNEJKakg6G/dk+a2UR7V2+hSxERKRolHQzJSdOpsBQt69cUuhQRkaJR0sFQ0RjOZdi6fmWBKxERKR4lHQz1zbMA6Ni4osCViIgUj5IOhsY99gagZ9OqAlciIlI8SjoYqqeELYb+VvWwKiKSNmIwmNm+ZlYZ3T/FzD5uZpPzX9o4KKtks02mvF1bDCIiablsMfwSSJnZfsAPgbnAz/Ja1TjaWrEHtV3rCl2GiEjRyCUY+t29DzgH+Ka7fwqYnt+yxs+2qj2Z3KseVkVE0nIJhl4zuwi4FLgzGleev5LGV1/9DPZkA63begpdiohIUcglGN4HnABc4+6vmdlc4Cf5LWv8JBv3os66WLtezUkiIpBD76ru/iLwcQAzawTq3f2r+S5svFQ3zQZg8+olMHuvAlcjIlJ4uRyV9ICZNZjZFOAZ4EYz+0b+SxsfjdPnAtD20PcKXImISHHIpSlpkru3AucCN7r70cCb8lvW+Jk0fT8AeqfOK3AlIiLFIZdgKDOz6cAFDOx8zomZnW5mC81ssZldmWH63mZ2v5k9ZWbPmtmZo1n+WLC6aXRSTXLr0vF+aRGRopRLMFwN3A286u6Pm9k+wCsjPcnMksB1wBnAwcBFZnbwkNk+D9zq7kcCFwL/NZrix4QZGytn0tCp/pJERCC3nc+/AH4Re7wEeFcOyz4WWBzNj5ndApwNvBhfPNAQ3Z8EFKRvis7avWjeuJC+VD9lyZLuJUREJKedz7PM7Fdmtt7M1pnZL81sVg7LngnEf4avjMbFXQVcYmYrgbuAj+VY95jqb9yHvVjH6k0dhXh5EZGiksvP4xuBO4AZhC/230bjRmIZxg29huZFwE3uPgs4E/ixme1Qk5ldbmYLzGxBS8vYX6P57jXVVFiKNStfHfNli4hMNLkEQ7O73+jufdFwE9Ccw/NWAvETA2axY1PRB4BbAdz9EaAKaBq6IHe/3t3nu/v85uZcXnp0Lq15FICtqxaO+bJFRCaaXIJhg5ldYmbJaLgE2JjD8x4H9jezuWZWQdi5fMeQeZYDpwGY2UGEYBj7TYIRTL7kJgB61i8e75cWESk6uQTD+wmHqq4F1gDnEbrJGFbU8d4VhCOaXiIcffSCmV1tZmdFs/0j8CEzewa4GbjM3Yc2N+WdNcykh3KSW5aO90uLiBSdXI5KWg6cFR9nZp8EvpnDc+8i7FSOj/tC7P6LwIm5Fps3iQQby2dQ17m80JWIiBTczh6b+Q9jWkURaK/dm+be1aT6x32DRUSkqOxsMGQ64mhCe7pjMnuxjlWbOgtdiohIQe1sMOx2P6uPq1xBrXXz2lLtgBaR0pY1GMyszcxaMwxthHMadivN7/giAJuWPlvgSkRECivrzmd3rx/PQgqtetZhAKTWvlDgSkRECksdA6XVNrE1OYW6LTrJTURKm4IhZqHvxYye19jWkyp0KSIiBaNgiGne9ygOsJUsWrOl0KWIiBSMgiGmfv0TVFkvq5Y8X+hSREQKJpdutzMdnbQi6op7n/EocrxMOf/bALQv15FJIlK6RuwSA/gGoVfUnxFObLsQ2BNYCNwAnJKv4sZbYo8DSZEg0fJSoUsRESmYXJqSTnf3/3b3NndvdffrgTPd/edAY57rG1/l1axgTxpaF9KvrjFEpETlEgz9ZnaBmSWi4YLYtN3u2zNZXsk8lvNqS3uhSxERKYhcguFi4L3A+mh4L+FynNWEbrV3K7VHv5vZifU898prhS5FRKQgcul2ewnwjiyT/zK25RRe47wT4VHYvOivcNJhhS5HRGTc5XJU0qzoCKT1ZrbOzH5pZrPGo7hCsJlH00eCnqWPFroUEZGCyKUp6UbCJTlnADOB30bjdk8VtWyuO4DDfBGbOnoKXY2IyLjLJRia3f1Gd++LhpuA5jzXVVD9fb0cnniVJ17bUOhSRETGXS7BsMHMLjGzZDRcAmzMd2GF1PiWf6LOuli+8IlClyIiMu5yCYb3AxcAa4E1wHnA+/JZVKFVzDkegJXPPljgSkRExt+IweDuy939LHdvdvdp7v5O4NxxqK1wGufQUT6Fv+t/mbau3kJXIyIyrna2E71/GNMqio0ZXdOP47jEizz66m7daiYisoOdDQYb0yqKUMOhb2GmbeQHv/pDoUsRERlXOxsMu11XGEOV738aACf4M7jv9m9XRGS7rGc+m1kbmQPAgOq8VVQsGmfTlpjEYT1P8tyqrRw2a3KhKxIRGRdZtxjcvd7dGzIM9e6eS3fdE1754edxfOIlPvYjnQUtIqVDV3AbRtWBb6HGupmfXFToUkRExo2CYThzTiJFggPaHmXxenXDLSKlQcEwnMo6+mafzFsTC7j0h2pOEpHSkNdgMLPTzWyhmS02syszTP9PM3s6GhaZ2ZZ81rMzKv/uLOYk1jG3fzkpXdVNREpA3oLBzJLAdcAZwMHARWZ2cHwed/+Uux/h7kcA3wFuz1c9O23e23CM47Y9yJnfeqjQ1YiI5F0+txiOBRa7+xJ37wFuAc4eZv6LgJvzWM/Oqd8D3/c0zi/7M3OnVhW6GhGRvMtnMMwEVsQer4zG7cDMZgNzgfuyTL/czBaY2YKWlpYxL3QkiSMvZk82su3le1jf2jXury8iMp7yGQyZus3I1kh/IXCbu6cyTXT36919vrvPb24uwKUg5p1JqnIy5yb/wg8f1rWgRWT3ls9gWAnsFXs8C1idZd4LKcZmpLTyKpKHncdbkwu45aHn2NqpHldFZPeVz2B4HNjfzOaaWQXhy/+OoTOZ2TygEXgkj7XsuhULqKKH8xMP8L+PLC10NSIieZO3YHD3PuAK4G7gJeBWd3/BzK42s7Nis14E3OLF3lPdhx+EOa/ng+V/4L/ufUlbDSKy28prn0fufhdw15BxXxjy+Kp81jCmOjayJxs53R7hm/fuyxffcUihKxIRGXM683k0PvIwNM3jI+W/58aHX+OVdW2FrkhEZMwpGEYjkYATP8E8XuNtycc593t/pV9nQ4vIbkbBMFqHXwjlNXy18ga6urp449cfKHRFIiJjSsEwWokknP+/1Pe38oHqB1i+uZNFalISkd2IgmFn7P9mqJrEZ/gR02wrZ33nL2zq6Cl0VSIiY0LBsDPM4AN/IpFIcNe839Od6ucN195PV2/GE7dFRCYUBcPOap4Hr/9Hpr52Bz9/Yxvt3X287qv3aWe0iEx4CoZdcdKnoLyGYx+9gi+dOpVNHT0c+2/30JvqL3RlIiI7TcGwK8oq4UP3gae45InzmT2pgg3tPXzkJ0+oWUlEJiwFw66adhCcfR3W3cqDFR9jzpRq7nlpPcd8+R7tkBaRCUnBMBaOeA8c/1FoW8MDFZ9iv+Za2rr7OOEr9/Li6tZCVyciMioKhrHyli9DTRNsWco9b17Prz96Iv3unPntP3Pyv99PsfcRKCKSpmAYK4kE/MOLUDkJbr+cIzb/kb9eeRqTq8tZtrGTw//1jyxe317oKkVERqRgGEtllfCp56GyAW7/EM3P/4CnvvBmrjnnUDp6UrzpGw9ywlfupbOnr9CViohkpWAYa1UN8OlFUDMV7v4cds8XufjYvXnsc6dx3tGzWLO1i0O+cDe3/G25DmsVkaKkYMiH8ir49CtwzAfh4W/Bv+9HU3kvXz//cA6eXo8ZXHn7cxz6xbv5+eMKCBEpLjbRdorOnz/fFyxYUOgycuMOD38T7rkKyqvhQw/AtANxdx5Y2MIVP3uSjp4UBsxqrOaOK06isbaiwEWLyO7IzJ5w9/k5zatgGAevPQQ/Phf6e2HqfnDFAjDbHhCfuOUpWrvCfoeypHHAtHp+9/GTMLMCFy4iuwsFQzFqWwu//CAs/TNUT4EP/wUmzdw++eW1rfz9D//G+rZuAGoqknzitP15++EzmDm5ulBVi8huQsFQrPr74TtHwualYAl44+fghI+FfRKRtq5e3vGdv7B8Uyfp/viSBrMaa7j58uOZoZAQkZ2gYCh2m5fC9afCto3hENd33QAHvi105x2zbGMHF//gMVZv2bY9JBIG0ydVc93FR3HYzEkkEmpuEpGRKRgmiiUPwM0XQW8n7HMKnP41mHZg5llb2rn7hXVcd/8rtHeHDvrKEsY7Dp/B6/dv4pg5U5jVWK39EiKSkYJhIkn1wYIfwt2fhf4U1O0B7/8DTNkn61M2dfTw0KIWvnTni2zq6CH9CRowuaacj75xP46a3cihMyZRUaYjkkVEwTAxdWyEB78Gf/vv8Piwd8Pr/zFcEGgYqX5n0bo2FizbzDf/tIhNnT3EP9KkwbSGKq4++1AO32sSzXWV2qoQKUEKhomsbS38z6nQuio83u9NcNxHYN9TQ39MOVjf2sUTyzbzr799kfVtXcQvKmeE/RSJhDFzcjXXnnc4B+xRx+QanT8hsjtTMOwOOjbAghvgoWsh1Qtl1fCWL8HhF0Fl3agW1dWb4tmVW/mnXzzD6q3bSPU7Q69AaoSwSBjMmFTNVWcdwt5Ta5jVWE1lWXLs3peIFISCYXfS1wMv/gbu/CT0RL2z1u0JF/0MZhy1w5FMuXJ3Vm/tYtG6Nr74m+dZvbWL/gyBkZZMGFNqKrjsxDnMnlrD3lNqmD2llkk15Tv5xkRkPCkYdlcrF8Bj/w3P3Roel9fAKZ8N+yPq9xiTl3B3NrT3sGxjB8s3dXLlL5+lN+WYhR4+sv21lCWMqXUVfOCkucyYXM3MaGiqq9QhtSJFoGiCwcxOB74FJIEfuPtXM8xzAXAV4TvnGXd/z3DLLOlgSOvaCs/fDn/6F+huC+OqG+HNX4J5Z0Lt1Ly9dEd3H8s3dbJsYycrNnVy7d0v05tyEkbWrY20proKzj1qFlNqK5hSW8HU2gqm1lUyNXpcU5HUjnGRPCmKYDCzJLAIeDOwEngcuMjdX4zNsz9wK3Cqu282s2nuvn645SoYhmhZBE//FB79HqRCdxpUTYJT/wUOfDs0TB/Xclq7elm9ZRurNm9j9ZZt/NcDr7J2axdO2I8x0l+bEVrHzIzGmnLOOXImU2oHwmNKXQVNtZU01pZTV1mmIBHJUbEEwwnAVe7+1ujxZwHc/Suxea4FFrn7D3JdroIhC3dY8wy89Ft45Dro2xbGzzomnFV94Nuhaf/C1khoqursSbGpo4cN7d1s6uhhY0cP37n3Fda2dm1vrnL3YZuu0rYHCZDycLtHQyUXHLM3DVVlNFSX01BVTkN1WbitKqe+qoy6qjLKkzrHQ0pHsQTDecDp7v7B6PF7gePc/YrYPL8mbFWcSGhuusrd/zDcchUMOWpZCC/eEbr9Tu+0LquG4z8M+781BEayrLA15qizp4+N7SFANnV0s7G9h82dPdz48FI2tHdvD5DUSG1ZGVj0TwgYCwETLac8aezRUMX/OXlf6irLqKlIUltZRnVFkqqyJFXlCarKk9GQoKosqf0pUrSKJRjOB946JBiOdfePxea5E+gFLgBmAX8GDnX3LUOWdTlwOcDee+999LJly/JS825r60p4+S64/8th/wSAJWHeGbDfaeFcicl7F7bGMZbqd9q7+2jd1ktrVy9tXen7fbR39fKDP7/GurYuILZT3R1n5H0luYg3myUsHUC2fRrAlNoK3nH4jO2hkg6YyrIklfHQKRsSQOUhmCrLE1SWJdScJjkplmDIpSnp+8Cj7n5T9Phe4Ep3fzzbcrXFsIu2bYYlD8Kr98GTP2L711dZNRz19zDnRNj7dVDXXNAyi4G709XbT1t3L+1dfXT2pOjsSdHR00d3b4qu3n66elNh6Ovnx48so6Wta3sg9KVC0CSjrYj4/7VcmslGK771M/B48FZQWXJwOGHGHvWVXPq6OVSUJShPJihLGBVlCcoSCcqTRnkyGh/dr4jdT08vS1o0PhqXSGjrqcgUSzCUEZqJTgMNoULtAAAQDElEQVRWEXY+v8fdX4jNczphh/SlZtYEPAUc4e4bsy1XwTCG3GHDIlh8Lzx4LXRtHphWXh1Oppt9YhjGeSd2qelN9dPdFwuaKHS6++IBFG6/c98rrIl26ONOb2rgSz/93zm+BQRjsxW0K+IREd+S2j4tttUTD7Jk0gY9Nx5ovX3hkrgVZQlmTKriI2/cj/KkbQ+0ssRAgJUlbHtoJRMD49Khtv05sfHJ3SzYiiIYokLOBL5J2H9wg7tfY2ZXAwvc/Q4L28D/AZwOpIBr3P2W4ZapYMijVC+sfhqWPRyGxfeCh55cKauCvzsvhMRex4VO/tSEMaG5O339Tm+qn95UuO1LpR8PjEvfv/SGx+ju7aeyPDFoayf+FdITfVmXlyUGBVN83lR/GF+WsIHpQ+ZNB1n6L6wYzrayIQ/ij7dfOyVDmBjQ1+9YfPqQ50PYwoSwbyutub6Ki4/fm/Io5F63bxPz9qzfufqLJRjyQcEwjvpTsPa5EBIPfT00Q8X/i1ZNhvnvh5lHwaxjx+wkO5HhuDup/rCl1NvfT29fLNj6nb4oyPr6o9tofDr40uNT/c63732FVVu2DV7+wAsBDNoii40eNP/2LRzLHGLbgy5L0uX6LTxnag0P/NMbc5x7MAWD5Ed/P7S8FM7AXv0kPPcL6OkYmF5WBQefHbYoZs2H5oOgTJ3zieysoaFWXREOQtgZCgYZP71dsPZZWPFYGBb+Afp7B6ZX1MGh74IZR8KMI2DaIQoLkQJQMEjhuMPm12D1U+GEu9VPw2sPDp6nohYOORemHx6GPQ4J40Qkb0YTDBPjDCeZOMzCjukp+4QtBYjCYimseToKjGfhmZvhqR8PPK+8Gg44I4TEHoeGrYv6PQvyFkRKnYJB8s8MpswNwyHnhHHu4WJEa56Jhmfh5d/BC7fHnwiVDeFoqOYDofkAaJoXAkNHRInkjYJBCsMMJs0Kw4FvGxjf1QrrXghbFutfCJ0EPnHTwGGzEM7annEkNB0Q+n9qnhfuN86BpK4PIbKrFAxSXKoaYPYJYUhzh/Z10PJyCIoN0fD8bZDqGfz88mrY97QQFlP3D8ExZR+omTK+70NkAlMwSPEzC81H9XvCPqcMntbVChteGQiLJ26ChXfBy3cOni9RFq54N3U/mLrvwO2UfbTjW2QIBYNMbFUNMOvoMAC86YvhNtUbdnhvfBU2Lh4YnrmZHU8nsrCcQ98FU/YNTVKNs2Hy7DBepMQoGGT3lCwPzUiZrkHR3Q6blkRhEQuOJ38E/X2D502UhSOlJs8eCIvt9/cOTVciuxkFg5SeyjqYflgYhurcFM7D2LIcNi+DLcvCZVTXPgfev+P8yXKYfsTg4EiHxqS9tDNcJiQFg0hczZQwzDx6YNzb/zPc9veHneBblsWCYym8+NvQTUimHm+SleGiSJP33jE46qdDYue6NxDJJwWDSK4SidD9eMN02Pv4gfFnXxduU33h3Iwty0N4pLc4Fv4+dESYad/GlLlR81Q8OOaEx7XNOl9DCkLBIDJWkmXhy71xNvD6Haf3dcOWFdEWRzo4lsMrf4QlD5Bxi6O8JoRQ/QxoSA8zB+5XNyo8ZMwpGETGS1klNO0Xhky622HrioEtjfRt62pY+pcdz9nYvtyq0FwVD4yGGaGpqmFm2PJIJPL3vmS3o2AQKRaVdTDtoDBkkuqD9rXQuiY0WbWujt2uhhd+BanuzM9NVoYOCxuisKifvmOIlFXm773JhKJgEJkokmUD3YhwTOZ5+vuhc0MUGFGAtK0ZCI9Ff4S+bZmfi4Wmq7mvz9BsFYVJZV2+3p0UEQWDyO4kkYC6aWGYcWTmedyhuzVzcLSuhq2rYPE9O57TkTZ1P6jbM1yxL31bPz06O326AmQ3oGAQKTVmUDUpDNMOzD5f77bBobF1ZThct21tuF31ZNgHkun8jor6EBQN02OhMSMWHlEXJ2q+KkoKBhHJrLx64Noa2bhDd1sUGGvCVkjbmhAebavD7fJHwtFYmY66qpk6sJURD42GGVA7Deqaw215Vd7epuxIwSAiO8+ifqaqGjJ3P5LmHs4qHxoa6cetq8PZ5e1rs7xOMpzzsT0s0kNT7H5zCBodwrvLFAwikn9mUDs1DHsemn2+VB90rA+B0d4S7revh46WsFXS3gLrX4JNv8u+DyS8YOg1d85J0T6XPaJQie6nb7UvJCMFg4gUj2TZwBFRI0n1hq2QjpZo2BDdRmHSvj5sibx6X/ZzQCCcBzL98FhoRMFRGw+RaSW1P0TBICITU7I8OiJqj5Hn7U+FEGlfFwuOdbHbdbBhcei+JNuWSKIsdMtel2HLIx4ktU0Tvg8sBYOI7P4SybBvoq555Hn7emJNV+szh8nLv4O+ruzLSJZD80FhR/r24GjecaiZUpQhomAQEYkrq4BJM8Mwkp6OgWar9JbH9kN614ed6UseGL4pKxFdO6RmatjaqGmKboc+bhq3IFEwiIjsrIracLTUlLnDz9ffD11bBvaHtK8fsk+kJZyxvva5cOXBbM1ZjfvAJ54a87cxlIJBRCTfEomBa300zxt5/vSO9c4NIUA6N0DHRphzYv5rRcEgIlJ8RrNjPQ/y2hevmZ1uZgvNbLGZXZlh+mVm1mJmT0fDB/NZj4iIjCxvWwxmlgSuA94MrAQeN7M73P3FIbP+3N2vyFcdIiIyOvncYjgWWOzuS9y9B7gFODuPryciImMgn8EwE1gRe7wyGjfUu8zsWTO7zcz2yrQgM7vczBaY2YKWlpZ81CoiIpF8BkOmXqyGdq/4W2COux8G3AP8b6YFufv17j7f3ec3N+dwgoqIiOy0fAbDSiC+BTALWB2fwd03unv6WoT/Axydx3pERCQH+QyGx4H9zWyumVUAFwJ3xGcws+mxh2cBL+WxHhERyUHejkpy9z4zuwK4G0gCN7j7C2Z2NbDA3e8APm5mZwF9wCbgsnzVIyIiuTH3DFdVKmJm1gIs28mnNwEbxrCc8aCax4dqHh+qeXxkqnm2u+e0k3bCBcOuMLMF7j6/0HWMhmoeH6p5fKjm8bGrNef1zGcREZl4FAwiIjJIqQXD9YUuYCeo5vGhmseHah4fu1RzSe1jEBGRkZXaFoOIiIxAwSAiIoOUTDCMdG2IYmFmS83suej6FAuicVPM7E9m9kp021jgGm8ws/Vm9nxsXMYaLfh2tN6fNbOjiqjmq8xsVex6IGfGpn02qnmhmb21APXuZWb3m9lLZvaCmX0iGl+063mYmot5PVeZ2d/M7Jmo5n+Nxs81s8ei9fzzqPcGzKwyerw4mj6niGq+ycxei63nI6Lxo//bcPfdfiCcef0qsA9QATwDHFzourLUuhRoGjLuWuDK6P6VwNcKXOMbgKOA50eqETgT+D2hU8XjgceKqOargE9nmPfg6G+kEpgb/e0kx7ne6cBR0f16YFFUV9Gu52FqLub1bEBddL8ceCxaf7cCF0bjvw98JLr/f4HvR/cvJFxPZrzXc7aabwLOyzD/qP82SmWLYaJfG+JsBnqe/V/gnQWsBXd/iNCFSVy2Gs8GfuTBo8DkIX1kjYssNWdzNnCLu3e7+2vAYsLf0Lhx9zXu/mR0v43Qj9hMing9D1NzNsWwnt3d26OH5dHgwKnAbdH4oes5vf5vA04zs0w9SefNMDVnM+q/jVIJhlyvDVEMHPijmT1hZpdH4/Zw9zUQ/vMB0wpWXXbZaiz2dX9FtHl9Q6yJrqhqjporjiT8MpwQ63lIzVDE69nMkmb2NLAe+BNhy2WLu/dlqGt7zdH0rcDU8a14x5rdPb2er4nW83+aWeXQmiMjrudSCYZcrg1RLE5096OAM4CPmtkbCl3QLirmdf89YF/gCGAN8B/R+KKp2czqgF8Cn3T31uFmzTCuWGou6vXs7il3P4JwaYBjgYMyzRbdFmXNZnYo8FngQOAYYArw/6LZR11zqQTDiNeGKBbuvjq6XQ/8ivCHui696Rfdri9chVllq7Fo1727r4v+g/UTrgeSbsYoiprNrJzwBftTd789Gl3U6zlTzcW+ntPcfQvwAKEdfrKZpXufjte1veZo+iRyb6Icc7GaT4+a8tzDNW5uZBfWc6kEw4jXhigGZlZrZvXp+8BbgOcJtV4azXYp8JvCVDisbDXeAfx9dGTE8cDWdFNIoQ1pZz2HsK4h1HxhdATKXGB/4G/jXJsBPwRecvdvxCYV7XrOVnORr+dmM5sc3a8G3kTYN3I/cF4029D1nF7/5wH3ebSHd7xkqfnl2A8GI+wTia/n0f1tjPce9UINhD3ziwjth/9c6Hqy1LgP4SiNZ4AX0nUS2jDvBV6JbqcUuM6bCU0CvYRfIx/IViNhM/a6aL0/B8wvopp/HNX0bPSfZ3ps/n+Oal4InFGAek8ibO4/CzwdDWcW83oepuZiXs+HAU9FtT0PfCEavw8hpBYDvwAqo/FV0ePF0fR9iqjm+6L1/DzwEwaOXBr134a6xBARkUFKpSlJRERypGAQEZFBFAwiIjKIgkFERAZRMIiIyCAKBhERGUTBIJIDMztiSHfRZ9kYdd9uZp80s5qxWJbIWNB5DCI5MLPLCCcGXZGHZS+Nlr1hFM9JuntqrGsRAW0xyG7GzOZYuFDM/0QXMflj1G1Apnn3NbM/RD3Z/tnMDozGn29mz0cXQnko6kblauDd0QVQ3m1ml5nZd6P5bzKz71m4SM0SMzs56kX0JTO7KfZ63zOzBTb44iofB2YA95vZ/dG4iyxcrOl5M/ta7PntZna1mT0GnGBmXzWzF6PeNL+enzUqJWm8T+fWoCGfAzAH6AOOiB7fClySZd57gf2j+8cR+r2B0G3AzOj+5Oj2MuC7seduf0y4QMothK4HzgZagb8j/PB6IlZLuvuKJKHjs8Oix0uJLs5ECInlQDNQRujm4J3RNAcuSC+L0I2ExevUoGEsBm0xyO7oNXd/Orr/BCEsBom6hn4d8IuoX/v/JlyBDOBh4CYz+xDhSzwXv3V3J4TKOnd/zkNvoi/EXv8CM3uS0M/NIYQrmA11DPCAu7d46O//p4SrzwGkCD2XQgifLuAHZnYu0JljnSIjKht5FpEJpzt2PwVkakpKEC7GcsTQCe7+YTM7DngbsP3auTm+Zv+Q1+8HyqLeQz8NHOPum6MmpqoMyxnuamBdHu1XcPc+MzsWOI3QW/AVhKuOiewybTFISfJwAZnXzOx82H7B9MOj+/u6+2Pu/gVgA6Ev+zbCdYx3VgPQAWw1sz0IF2JKiy/7MeBkM2sysyRwEfDg0IVFWzyT3P0u4JOEi+CIjAltMUgpuxj4npl9nnDd3FsIXZ7/u5ntT/j1fm80bjlwZdTs9JXRvpC7P2NmTxGalpYQmqvSrgd+b2Zr3P2NZvZZwvUADLjL3TNdf6Me+I2ZVUXzfWq0NYlko8NVRURkEDUliYjIIGpKkt2emV0HnDhk9Lfc/cZC1CNS7NSUJCIig6gpSUREBlEwiIjIIAoGEREZRMEgIiKD/H+UnKiR86daZQAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x16592ab3240>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "#调整max_depth和min_child_weight之后再次调整n_estimators(6,4)\n",
    "xgb2_3 = XGBClassifier(\n",
    "        learning_rate =0.1,\n",
    "        n_estimators=340,  #数值大没关系，cv会自动返回合适的n_estimators\n",
    "        max_depth=4,\n",
    "        min_child_weight=1,\n",
    "        gamma=0,\n",
    "        subsample=0.7,\n",
    "        colsample_bytree=0.8,\n",
    "        colsample_bylevel=0.7,\n",
    "        objective= 'multi:softprob',\n",
    "        seed=3)\n",
    "\n",
    "modelfit(xgb2_3, X_train, y_train, cv_folds = kfold)\n",
    "from sklearn.model_selection import cross_val_score\n",
    "results = cross_val_score(xgb2_3, X_train, y_train, metrics='mlogloss', cv=kfold)\n",
    "print(results) \n",
    "print(\"CV logloss: %.2f%% (%.2f%%)\" % (results.mean()*100, results.std()*100))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 76,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "D:\\anaconda\\lib\\site-packages\\ipykernel_launcher.py:1: FutureWarning: from_csv is deprecated. Please use read_csv(...) instead. Note that some of the default arguments are different, so please refer to the documentation for from_csv when changing your function calls\n",
      "  \"\"\"Entry point for launching an IPython kernel.\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAA2EAAANGCAYAAAB0pYaxAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAAPYQAAD2EBqD+naQAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMS4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvNQv5yAAAIABJREFUeJzs3XecVNX5x/HvQxcpChZEReyKDTsqKoqKxm7ssfCza2JPjJpEiS2WiCV2jb1iF42ColgQC9jFggVEQBSlSofz++PePee6O7s7uztzp33er9e8/O6du3fO7C64D+fc55hzTgAAAACAdDQr9AAAAAAAoJJQhAEAAABAiijCAAAAACBFFGEAAAAAkCKKMAAAAABIEUUYAAAAAKSIIgwAAAAAUkQRBgAAAAApoggDAAAAgBRRhAGoGGZ2p5nNN7ONMjx3rpk5M9u72vEO8XNvm9l0M1toZlPM7AUzO9zMWifO7R5fI/mYaWYfmtkZZtY8jfdZFzM7xcz6F3ocTWFmbc1sgJn1yfBc//jr3r0A4+oaj6tn2q9dSGZ2t5nNLvQ4JCn++jszW67QYwGAulCEAagkZ0j6QdI9Ztay6mBclP1T0t3OucGJ42tLel/S3yS9IekoSTtLOlXSREl3Svp7htf5j6Rt4sfBkkZIukbSlbl/Sw12iqT+hR5EE7WVdKGkPhmee07R131ymgOKdVU0rooqwgAADdei0AMAgLQ452aa2bGShioqni6Mi7H7JE1RVKRJksyshaSnJHWStJVz7rNqlxtkZhdJ2jTDS33nnHsr8fELZrahpMMknZ2zN4QanHM/Sfqp0OPIJTNr65ybU+hxAAByh5kwABXFOfeSpFsknW9mm0saIGkTScc652YkTt1fUg9Jl2YowKquNd4591SWLz1D0sLkATNrZmbnmNnn8TLJH83sXjNbpfonm9kx8bLGeWb2i5k9aWbrVztnDTN72MwmxdebYmbDqpbHmdk4SRtI2jGxXHJcXYOOz7nBzI40s8/MbE48jr2yfN/Ja7Uys78n3u9PZnaXmS1f7bydzWy4mf1sZnPN7Dszezxehthdoci6MPE+7o4/t8ZyxPhan5jZNmb2ZnzNcWb2f/Hze5rZe/F7+9jMdq82nrXicY6Nz5loZoOTy1rjpZHvxh/elRjXgMQ5+5jZyPgas8zsRTPbptprVS2n28zMHjOzaZK+jp+r8/tby9f8jPh6a2V47gozW1C1dM/MNjWzZ+Ofw/nx6zyX6eexMcysdzzeWfHX4E0z27OW80bGP+sTzexiMzuu+ve1iWPZ0MyeNrNp8et8YGZHVzunWfzz+kX8MzPdzD4ys9MT5yxvZreZ2YTEz/QIM9slF+MEUL6YCQNQif4iqZ+kxyStKukW59yL1c7ZNf7vM424fjOLZtIkqaOkfSXtLumKaufdLOkESTdIelZSd0kXS+pjZps556ZKkpmdJ+kySQ9JOk9SZ0XF40gz29I5Nza+3v8kNZd0jqTvJC0naVtJy8TP7x+/5xmKliVK0vws3s+ekraUdIGk2fH1nzSzdZ1z32Tx+TKzZpKelrS9omWZb0paTdEy0OFmtoVzbm78S/Zzkl6XdIyk6ZJWVvT1a6VomeHukl6Q9F9Jd8QvUd/sVxdJd8Wv/b2iJaV3mtmqkg5U9PWdEb/Hp8xsDefcpPhzu0r6WdK58et0knS0pLfNbFPn3BeS3pP0f/FrXBK/B8WvJTM7XNIDimZhD5PUOv46Djezvs65N6qN9wlJDyv6B4Ol42P1fX8zuV/Rz11/JZbOWnR/4hGSBjvnpprZ0pJelPStpD8qmhnuImknSe3ruH5WzGzH+PofSTpW0c/dKZIGm9lhzrlH4vM2js/7UtHXeI6kk+Kx5oSZravo5+9HSacp+t4eIeluM1vROVe1bPgcRX/OLpH0mqSWktbTb7/e90naTNGS5S/j5zZT9GcUAGrnnOPBgwePinso+kXYKfqlvl2G55+Pn29d7bgp+gesqkfzxHPd48/J9Lir2rnrxcdvrHb9reLjl8YfL6PoF9Hnqp23qqR5kh6IP+4cf97p9bzvTyQNb8DXySm6j6594tiKkhZLOrcB1zk0vtYB1Y5vER8/Of749/HHm9RxreXicwZkeK5//Fz3xLHh8bHNE8c6SVoUf227Jo5vEp97ah2v31zRL+RfShqY4b30r3Z+M0X3EH4kqVnieDtFxc6IxLEB8TX+We0aWX1/axnv45ImVHvtPeLr7RV/vHn88b6NuP7dkmbXc87I+L22SxxrLunjeGwWHxukqNBfrtrX79Pq39daXqfq67dcHec8FP/ZWbXa8f9J+lVSx/jjwZLer+f1Zkm6pqFfMx48ePBgOSKAihPPypwqaYmkFRT94p2t0xUtK6x6fJjhnOsUzRxtqWgm4XxFDToeSpyzU/zfu5Of6Jx7R9JnkvrGh7aRtFSG8yZIejlx3i+Klq39xczOipeW5erv+Fecc7MSrz1F0SzCag24xl6KZrUGm1mLqoekDxQVeX3i8z6QtEDSbWZ2tJmtkYs3IGmyc2501QfOuV8UvYcPXJjxkqKvvZR4b/FYzzezMWa2QFHxtkDS2pJ+syS0Fusqmk27zzm3JDGG2YoKpF5m1rba5zxe7eOmfH/vkrSKpOQSuf9T9HV/Pv74K0nTJF1hZieZWY8sr12veJZta0mPxe9ZkuScW6xoJmkVRV8jSdpR0ssungWOz1uiqDjLlZ0lDYv/DCXdrajpS9US0XckbWJmN5lZPzPrkOFa70jqHy9b7GWJhj8AUBeKMACV6M+KftE6XNJYRcvSlqp2znfxf6sXGg8qFFjv1XL9751zo+LHcOfcvxQtMzzIzPrF51QtV8rUxW9S4vmsznPOOUUF2RBFy6jek/STmV1vZk1dTvZzhmPzFRWH2VpR0azeAv22iF2oaNnbcpLknPtaUbHwo6QbJX1tZl8n78NppF8yHFtQ/bhzbkEc2yQOD1T0/XtK0t6KCootFRXg2XwN6vseNpO0bLXjvzm3id/f5+PrVd0Dt6ykfSTdGxdCctH9kDsqKoIvk/RpfE/YP3NQWCyraAa5tvcv/fbnfUqG8zIda6zOWY7lX4r+ruil6Gv4c3xP2xaJzzlE0j2SjlM02/eLRfd1dsnheAGUIYowABUl/hf+ixT9AvqIouVra0m6tNqpVfeI7ZM86Jz7sarAUrQUKVsfxf+tmnWrKmxWynBuV0lTG3ieXNQo5FjnXBdFMwvXKLrv5qoGjDNfpip6L1vW8qi6R03Oudedc3srup+ul6Jfbq81s0PTHnTsCEU/L+c754Y4596Jv//Z7kVV3/dwiaJZqCRX/cTGfn8TM077mdkyiv7xobWiGbLkeR875w5VVIT0lPSIonvkmtrRc5qi91jb+5d++/O+YobzclnU/JzNWJxzi5xzA51zmylavnqYomXAQ6pmLp1zU51zZzjnuiv6B5vzJB2gajPXAFAdRRiAihEvf7tH0S9Zp0uSi1rJD5R0upltlzj9SUljFHVRXC8HL1/Vwe7H+L8vx//9TcMBM9tS0RK3YfGhkZLmZjhvFcXLqjK9mHPuS+fcJYruudks8VRDZ7By5VlFv9w3T8wSJh9fVP8E59xi59zbihpFSOF9VDUTSet9OFVrYBJ39Vu52nm1jesLRfeEHW5mlrjG0orugRvpGtiCvo7vb23uUjS7d5iif3gY6Zz7vJZrO+fch865MxUtIc3m+nWN9VdJb0s6IDnjHC+nPEJR85Iv48OvStrZEpstx+cd1JQxVDMsfo2u1Y4fpegewbeqf4Jzbrpz7jFFs7OdFN3/Wf2c75xzNyj6B5wmfc0AlD+6IwKoJOcpap6wh3NueuL4PxQtM7vTzHo65+Y65xab2X6Kln+9Y2a3K2rwME3RsrqtFc1qZWpf383MesV5aUVLH8+TNF5R1zs5574ws9sknWpmSxQtd+quaNnbBEWzHHLOTTeziyVdZmb3KrqvrLOiTYHnKeouWNVV7gZJjypaYrlAUZG2saTLE2P7WNKhZnaIpG8kzXPOfdyQL2IjPSzpD5L+Z2bXKbqXZqGi+4F2kvS0c+5JMzspHvdzipaEtlHUJVGSXpIk59wsMxsvaV8zG6ZoSeFU59y4PI39WUX3/XyuaEZzc0UdNr+vdt7XigrmP5jZZ4oaTExyzk0ys3MUdUd81sxuVTQT9RdFP0vn1jeABnx/M3LOfW5mIxX9HK6qqCtn8vp7KZpVe0rRz4UpmtFZRmFWuC7NzezADMd/dc49H7/ui5JeMbN/x+M/RdKGkg6Ll1tK0Yz03pKGmdmlir6eJyl0iFyi7OxtZjVmquNC6p+K7lF8xaK9/n5R9LO5p6Rz4qWZMrPBihrZjFLUFXM1RXsJjpc01sw6SnpF0RLlzxXNjG+pqHvnE1mOE0ClKnRnEB48ePBI46GoYFog6bZanu+lqOPfwGrHOyj6BfIdhb2+pihqNX6KpLaJc7urZlfEuYpmQq6R1KXatZspur/ni3hsPyluVJBhfMcqugdpvqLZiack9Ug8v4Ki2Y6qX/5nxeefod92ZVxNUWE5Mx7fuHq+bk7SDRmOj5N0dwO/By0ULW37IP66zIrHe4uktRLfhyfi689TNGs5XNLe1a7VV9F9UfPiMd4dH++vzN0RP6nlPTxb33tWVIjcEX/ff1XUPr93fN3h1T730Pg9LVC1Do6Ktip4K37vsxUVldtW+/wBytDdL9vvbz1f/+Pja8+R1KHac+sqKia+ip+frmj26ugsrnu3au8KOi5xXm9Fs1Cz49cYqbg7Y7Xr9Y6/TvMU3bt1paI/J05x58I6xjKgjrG4xHkbKtp+YrqiP1MfqGZXy7MkjVD053K+ouLrDkmrxc+3VrTNxIeK/m6Yo6gYG6DE3ws8ePDgkelR1RIWAACgKJnZUEWF9TqFHgsA5ALLEQEAQNEws4GS3le0LLeToqWCuyqaDQaAskARBgBokrjhSV2WuMT+WEA9mivqYNpF0TLCMZKOdM7dX9BRAUAOsRwRANBoZtZd0rf1nPZP59yAvA8GAIASwUwYAKApJinqCFffOQAAIMZMGAAAAACkiM2aAQAAACBFLEfMwMxMUldF+7AAAAAAqGztJU1yOVpGSBGWWVdJ3xd6EAAAAACKxiqSJubiQhRhmc2SpAkTJqhDhw6FHgsAAACAApk5c6ZWXXVVKYer5CjC6tChQweKMAAAAAA5RWMOAAAAAEgRRRgAAAAApIgiDAAAAABSRBEGAAAAACmiCAMAAACAFFGEAQAAAECKKMIAAAAAIEUUYQAAAACQIoowAAAAAEgRRRgAAAAApIgiDAAAAABSRBEGAAAAACmiCAMAAACAFFGEAQAAAECKKMIAAAAAIEUUYQAAAACQIoowAAAAAEgRRRgAAAAApIgiDAAAAABSRBEGAAAAACmiCAMAAACAFFGEAQAAAECKKMIAAAAAIEUUYQAAAACQIoowAAAAAEgRRRgAAAAApIgiDAAAAABSRBEGAAAAACmiCAMAAACAFFGEAQAAAECKKMIAAAAAIEUUYUVszoJF6n7uc+p+7nOas2BRoYcDAAAAIAcowgAAAAAgRRRhJYJZMQAAAKA8UISVIAoyAAAAoHRRhAEAAABAiijCShyzYgAAAEBpoQgDAAAAgBRRhJURZsUAAACA4kcRBgAAAAApoggDAAAAgBRRhJUpliYCAAAAxYkirAJQkAEAAADFgyKsRNz66jeFHgIAAACAHKAIK2JLljifb3/925xck1kxAAAAoLAowopYs2bmc+sW4Vv17EeTCzEcAAAAADlAEVYi7jlmS58HPDPG58WJ2bKGYlYMAAAASB9FWIlYa4V2Pi/btqXPx987uhDDAQAAANBIFGElaPCp2/n80fczfGZWDAAAACh+FGElqE3L5j73WqOTzyfex6wYAAAAUOwowkrcfw7r6fMHE8KsmHPMigEAAADFiCKsxJmFDopbdl/W57MGfZST61OQAQAAALlFEVZGbvzDpj6/PnZqzq9PQQYAAAA0HUVYGWmWmBXr1mkpn68f9lUhhgMAAAAgA4qwMnXfsVv5fO/I8Tm/PrNiAAAAQONQhJWppVu38Hn59q19vvDpT3P+WhRkAAAAQPYowirAoyf28vm5j3/wecGiJYUYDgAAAFDRWtR/CgqlbasWGnf5npLUpBmmdm3Ct7nHSu01ZvIsSdLvbx7ZtAFmMGfBIvW4YIgkacxF/dS2FT9iAAAAQBIzYRXmrv/b0ufJM+b5POyzH3P+WixTBAAAAGqiCKswzZuFDoqn9V3L578+/nFeX7fHBUMoxgAAAABRhJWMqqWJ4y7fM2dL/I7aZjWfl2rZ3OerhnyRk+sDAAAAqIkirATloyB7+k/b+vzIu9/77JzLyfWrsEQRAAAAlY4iDJKkTku38nmFREv7o+98txDDAQAAAMoWRViJy8es2MMnbO1zVSdFSXrty59ycv0qzIoBAACgElGEoYYOS7X0+aQd1/D5rEEf5e01KcgAAABQKSjCykg+ZsWO2351n1s2D50VT7rvvZxcPxMKMgAAAJQzijBkbdCJvXweNX6az8O/yO0yxSQKMgAAAJSb3EyXoOhUzYpJylnxsmqntj4fve1quufN8ZKkPz+av2WKAAAAQLlhJgyNcurOYaPn5B5jJ9/PMkUAAACgLhRhFSAf94olPfXHbXx+d1xYpvj0B5Ny/lpVKMgAAABQqijC0GSd24V9xfbZZCWfL372s1Ren4IMAAAApYQirMLke1bsgr17+NwtcQ9ZlZ9nz8/5ayZRkAEAAKDYUYRVsHwXZI+f3KvGsX1vfDPnr1MbCjIAAAAUI7ojQlJ+uimaWY1j8xYu8fnON8bl5HUAAACAUsJMGFJ17SGb+HzT8K9Te11mxQAAAFAsKMKQqt5rL+dz53atfD7+ntGpjYGCDAAAAIVEEYYa8n2vWJXHTw6t7d+fMN3nW1/9Jm+vWR0FGQAAANLGPWGoUz7uFavSrnX48fvD1t30wNvfSZJuf/1bf/yfg8fk9DUBAACAQmMmDEXhzF3X9nnXHiv4PPjDyT4//M6EvI6BWTEAAACkgSIMWUtrmeK/DtjI5317dvX530O/9Hkq+40BAACgRFGEoVHSKsj+sdf6Pq+67FI+7/2fEXl7zeooyAAAAJBLFGEoGY8lGnksXOx8PvORD1MbAwUZAAAAmooiDE2W1qxY82Zh8+c7jt7c59fHTvX5fx9PVlooyAAAANAYdEdETuWzm2JSz1WX8fmAzVbWE+9NlCRd8HTopvjTrPzeN5Y0Z8Ei9bhgiCRpzEX98lqMAgAAoLQxE4a8qSrIxlzUL6+vc/7v1vN5hfatfd4rcd/YosVL8jqGpB4XDGF2DAAAALWiCENZeeZP2/q8eEm4b2yXga/77JxTGliuCAAAgEwowpB3ad0zJkktmocf6fuP28rn2fNDEXTUne/mdQyZUJABAACgCkUYUpVmQbZel/Y+3/1/W/j82eRZPk/4ZU5ex5BJsiCbOnsexRkAAECFoQhDRdhw5Y4+H7bVqj4fetvbhRhORsyWAQAAVAZauKFg0uqkWN3Zu62jh96ZIEmavyg07Lj2pbGpjaE+dFsEAAAoX8yEoSikuUwxacA+PXy+/63vfP5wwvTUxgAAAIDKQhGGopNmQbbXxiv5vPN6y/t87D2jfU6rm2JtuIcMAACgvFCEoailWZBdeeDGPjezcDy531iy7X2hcQ8ZAABAaaIIQ8lIsyB78PitfZ4yc77Pe1wX9htbmOIG0PWhIAMAACgdFGFABmut0M7nqw8OM2S//LrQ592uSX8D6GxQkAEAABQ3ijCUpDRnxXZcJ9wrdsPhPX2eNS8UOMXU6j6JggwAAKD4UIQBDdBrjc4+337UZj5//dOvNc79Yca8VMaULQoyAACA4kARhpJXqPb2m3Zb1uezdl27xvPJhh5f/zQ7lTFli4IMAACgcCjCUFYKVZAdvnW3GscSDRZ1yK1hueL30+amMKLsUZABAACkqyiKMDM7xcy+NbN5ZjbazLav5/xlzOxGM5scf85nZva7plwT5adQBVmVF87o7XOr5uGP2gE3vZn6WLLFnmQAAAD5V/AizMwOkXStpEslbSrpdUnPm1nNqYXo/FaSXpTUXdKBktaVdLykiY29JspfIQqyzu1a+/ziWeHfAJJbjR1157upjKWpmC0DAADInYIXYZLOkvRf59wdzrnPnHNnSJog6eRazj9GUidJ+znnRjjnxjvn3nDOfdiEa6KCFKIgW7p1eJ0Hj9/K5zGTZvp85iMfqhT0uGAIxRgAAEATFLQIi2e1Npc0tNpTQyVtW8un7SNppKQbzWyKmX1iZuebWfPGXtPMWptZh6qHpPaNe0dA/dZZMfx4nbFLaOjx+tipPh9/z+hUx9QYzI4BAAA0TqFnwpaT1FzSlGrHp0jqUsvnrKFoGWJzSb+TdImksyX9rQnXPE/SjMTj+6zfAUpaoe8bO6JXWCF70o5r+Pz+hOk+H3fPqFTH1BgUZAAAANkrdBFWxVX72DIcq9JM0o+STnDOjXbOPazo3q/qSw0bcs1/SeqYeKyS5bhRRgpdkB23/eo+/7nfOj5/MGGGzyff/57PztX241xYFGQAAAB1K3QRNlXSYtWcoVpBNWeyqkyW9KVzbnHi2GeSusRLERt8TefcfOfczKqHpFkNextAbh265ao+/2nnNX1+d9w0n5P7kP00a346A2sgui0CAADUVNAizDm3QNJoSbtWe2pXSbX18R4haS0zS459HUmTnXMLGnlN4DcKPSuW1H/b7j7/ebcwQzZlZii89rjuDZ8/mRhmzooVs2UAAKCSFXomTJIGSjrOzI4xs/XN7BpJ3STdIklmdq+Z/Stx/s2SOku6zszWMbM9JZ0v6cZsrwk0RDEVZIduFWbIrj+sp8/NEjtD978r3EP23S9zUhlXU1CQAQCASlPY3yglOeceMbPOki6QtJKkTyT9zjk3Pj6lm6QlifMnmNlukq6R9JGi/cGuk3RFA64JNEpVQSap4AXDtmt29vmVP++oHa96VZLUukUzzV8U/ZE54KaR/pyps4tzyWLSnAWL1OOCIZKkUX/vqy0uGSZJGnNRv4IXwAAAALlSFL/VOOduknRTLc/1yXBspKRejb0mkAvFVJAl9yEbcsb26vPvqCBr3sy0ON4deq/rwz1kxdrUozbJ4oyCDAAAlLpiWI4IlLyqgmzMRf0KPRS1axMKlJfO2t7nRUtC4XXY7e+kOiYAAAAEFGFAGWvfpqXP9x+3lc9f/Tjb51MffD/VMTUV95ABAIBSx5oeIIeSSxQlFc1yRUlar0t7n8/ebW1dPXSsJGnkN7/448fcHZp6LCmBJYvcQwYAAEoRM2FACoqpw6IkHbZVN5//tFPYh+yj70N7+10Hvu7zlJnz0hlYjrA/GQAAKGYUYUCF679dd5//uvu6Ps+Yu9DnPRNNPUoZSxkBAEAxoAgDUlZss2JJB22xis939t/C5+Q+ZFWe/WhyGkPKG2bLAABAoVCEAcho41U6+jz0zB1qPD/gmTE+P/thaRdkST0uGEIxBgAA8qq4/hkeqDDF3MgjaZm2LWsc69ZpKX33y1xJ0oDBoSC77H+fpzaufGJvMgAAkC/MhAFFqJiXLFZ5/ORtfE7Omj3x3kSfrx/2Vapjypfali6yjBEAADQGRRhQ5Iq1IDMLN4ol7x/bf9OuPt87crzPV7zwhc/zFi7O8+gAAACKF0UYUEKKtSBL+tue6/vca41OPj866nuf+179ms/JjaNLGY0+AABAtijCgBJVCgXZDYdv6nP/bVfzef6iJT4fetvbPt/22jfpDCxFtMUHAADVUYQBZaAUCrI/7byWzw+fsLXPbVs19/m21771+bBEcfbr/PIoXijIAACARBEGlJ1SKMjWWqGdzy//ObS/32PDLj6PTSxT3GVgWL74/S9z8jy6dLB8EQCAykURBpSxUijIWjQLfw1dvN8GPl954EY+L1zsfN7vppE+D/10Sp5Hlz5mywAAKH8UYUCFKIWCLGnn9VbwedCJYflii2ahK+P5T37i8wGJ4uyhd77z2blQwJUaZssAAChPFGFABSq1gmyN5TMvX1w7sazxu8QyxauHjvW59xXDfX7uo8l5GmG6KM4AAChtFGFAhSu1giw5xocSDT5uPXIzn7dfezmfk50YL3xmjM/JlvnlguIMAIDSQBEGoCxsvtqyPl9zyCY+P3nKNj73WKm9z8nNoy959rM8j66wksUZRRoAAIVHEQbAK7VZsWys2qmtz/ceu5XP63YJBdlTH0yq8XnvfzctvwMrIjQDAQAgXRRhADIqx4Is6YHjQkF2zHbdazx//L3v+XzOYx/5PGPuwryOq9BY0ggAQP5RhAGoV1VBNuaifoUeSl6cstOaNY6t2KG1zy9//pPPfa8Oe5a98vmP+R1YEamtOKNQAwCg4SjCAGSt3GfHkp47rbfPZ+yyVsZz/vLYxz6XY6OPhqJQAwAgOxRhABqlkgqyI3qt5vPQM7f3uVvifrNko49bX/0mnYGVIJY7AgAglfdvTgBSUVWQVanK5fiLdaelW/n8xCnbaItLhkmS1l+pvT6bPEuSdPvr3/pzDrolbCI9s8zvJ2uKOQsWqccFQyRJYy7qV/aFPQCgsjETBgA5cF+i8+KeG3Xx+dupYRPpXQe+7nMldV9sKGbLAADljn9qBJA3lTRDlvTPfTfQcx//IEn690Eb68+PRt0VFzvnz0l2X7z8+c99dolz8FvJ2bJRf+/rZyGZOQMAlBr+rwUgdZVUnPVZd3mfn/nTttrnhjclSZ2XbqWff10gSXps9ER/zn43vulzubfDz5VkcVaFIg0AUMxYjgigaJR7s4+uyyzl8wtnhO6LR27TzeeJ0+f5nGyHf92wsT5/P21uvoZYlujaCAAoNhRhAIpSuRdkZubz6X3X9vmqAzfKeP59I7/zOTlb9qcH3/d5WjyzhobjPjQAQJrK7zcbAGWntuWLteVS/sV5p/VW8HnY2Tv42bBDt1xFD78b7UXWqnkzLVi8RJL01je/+PN3uyY0/rgicZ/ZrHksa2ws7kMDAOQDM2EAyk5yFm25dm1Kdkat41Ktxms8AAAgAElEQVQtff5zv3V9fuPcPj7/Y6/1fU629Hg0cZ9ZclljlcdGs7l0UzBzBgBoCoowABWjXJY4NkssZdy3Z1efnz893Gd2XO/uPi/J0HDx8ufD5tL73hCWN344YXqORlmZuP8MAJANijAAFalcCrKk5du39vmkPmv6nCzOqmy3VmefJ04PjT6OvWe0z+c/8YnP3/8S9jtD0zCLBgCgCANQ8aoKsjEX9Sv0UPIiWZxVue7Qnj7fcHjInZdu5fPQMVN83u+mkT5f+txnPo/8+uecjbPSUZwBQOUoj3/+BYAcqKT9y5J6rRFmxV44o7e2vPRlSdLpfdfSdcO+kiS1bG5auDha1/jk+5P8+ac+9IHP/RKNQYZ+Ggo4NE1tzUGq0CQEAEoPM2EAUI9yXLpYm2Tr/CO3Wc3n4X/Z0efjt1/d57VWaOfzz4kW+ec/GZYyJpc1/jq/vAvaQkjOoJX7PxgAQLmgCAOABqikgiypdYvmPp+44xo+P3zC1j7fe8yWPq+zYijOkssa+1z1qs9XDQnNQSZNZwPqXMimMUhDM4UdAOQeRRgANFKlFmS16dG1g88PHh+Ks1MSTUKSjRofeTe0yd8n0aHxumFjw/kuQ2tHpIp71QAg9yjCACAHKMhqd0yiXf4LZ4ROjf+3XTjeollYBnnfyO98/t31I3x+7cuf8jNANArt+AGg8SjCACDHymWz6HxYrl3o1PjHncIM2et/7ePzgZuv7PNPs+b7fNagj3xOLmv8y6PhOIoLs2gAkBm/EQBASpLdF/kl9LdaNg//JnjuHuvpsdETJUm3HLGZTrr/PUnS2iu009gfZ0uSZicafLzyRc0ZsuQG1K8yg1Z0auv4SPdHAJWCmTAAKABmy7KzRfdlfX4o0QTksZN7+XzuHuvW+LzkBtRnJ2bQjr7zXZ+f//iHnI0T+cNsGoByRBEGAEWEe8uy073z0j4fuPkqNZ6/5YhNfU52avx00kyf//H0pz7//uawGfULn4TibNGSJU0fLPIiV50gKeYAFAJFGAAUKWbLGm+L7p18TnZqvPLAjXzerNsyPo//eY7Pf38qFGfJe88eeic0DFm4mOKsXDDTBqAQKMIAoMRQnDXezuut4PNtR23u87WHbOLz5quF4mzewlBsXT00tM7f7vJXfE7udzZr3sLcDRYFRXEGIJ8owgCgTFCcNV7vtZfz+dYjQ3H26Enh3rOd11ve5yWJ7cuS+53t9O/XfB6eoWEISh+t+QHkAkUYAJS5ZHFGYdYwqy8X7j278sCNfU7ud3bkNt0yfu6fE63zbxn+tc8LFrGUsdwlCzUKMgCZ8H9hAKhAtMtvmuR+Z6f3XdtvMD3s7B3U9+poNqxLhzb6YeY8SdIdb4zz52+bWMp4xfOf+zxv4eJ8DhkFUls7/iq04AcqE3/qAaDCJQsySRkzhVp2Oi7V0ufBp26rLS99WZK098YrafBHk2uc/2i8H5ok7XhlaAIyaNSEPI4SxSSbPdNqyxRwQOliOSIAoF7cb9ZwZubzhfv08PnFM7f3+ahtVvN5sQs3ml35wpc+73n9Gz7f9EpY1giw7BEoXRRhAIBGozhruGWXbuXzaX3X8vl/p23n83ZrdfZ5ysz5Pt85YlyN653/xCc+z1/EksZKRTdHoLRQhAEAco5NpxtuhQ5tfL7u0J4+33F06NZ40OYr1/i8oWOm+Lzd5cN9Puq/7/h8ygPv+/zg22G/s4nT5jZ+wCgJzJYBxYn/MwIA8qq2e874hTA7PVcN+5b9dY/1fnMfmSSdsMPquu21b2t83pjJs3x+59tffB74YtjvbN8b3/R5l4Ghvf7VQ8NyyFc+/9FnZtpKW0PvP+OeMyB/mAkDABQESxlz44Qd1vD5pbPC/WbXHho2oL543w183imx31mbluHXgOlzwkbTD70TGoP85bGPfU7OtP07sUn1p5NmNmboKHIscQTyhyIMAFBUWMrYeMu0Dfeb9V4rbEC9x0ZdfL4qsd/ZG3/dyef7j9vK56O3DQ1DNl6lY8bXejixSfXRd77r86G3ve3zC5/8kPXYUTrYsBpoOoowAEDRqm22LFMec1G/Qg+3pK3Xpb3Pp+4cGobc2X8Ln186awefD9tqVZ9X7BD2Tfvqx9k+//2pT33eNbHc8b9vhOWTP8yY15Rho0j1uGAIRRpQB4owAEBZYAYt/5ZpG/ZBO3u3dXx+7rTePv/7oDDTtvlq4X62aYnljjcP/8bnvf4zwufj7xld4zXH/fxrE0aMYsVSR1Q6ijAAQNnhfrPC6bNuuOfs1iNDZ8d7j9nS5703XsnnFs3CfmrvT5he43oH3vyWz/veEBqJDHwxNA+ZPY9f3MsFxRkqBUUYAKBiMFtWOD26dvA5uXn1iHPDfWmX7r+BqmvbqrnPE6eHlvoPvh2ah/T596s+nz3oQ59fHzvV589/CN0ilyQ2xkZpoDhDuaEIAwBUJGbLikPzxExYvw261Hj+1b/s6HPy/rSDt1gl4/Ve/TIUXmc+EgqyI+4I+6Zt869XfE5uds3Sx9JDkxCUKv5PAwBAQm37miUzv9SlxywUaclOjefsvq4GjYo6NL7x1z7qfcVwSdLZu62tq4dGe6H1WKm93y9tuXatNHX2AknS4iVhJiy52XVy6eM/B4/x+a1vfs7V20EBZLM/GnuiIW3MhAEA0EDMohWXNi3DksXDturm873Hhrb7L5wR9lB79tTtfD5jl7V9bt0i/Fo0+MPJPv/pwQ98Pvz20IL/4XfDkshnE+d/Ppl900pNNjNqycyMG5qKIgwAgBzhnrPS0KVjG5+P6BWKtuGJpY8n9wmbYK+x/NI+fzkltOD/95DQHGRAYubsiP+GfdP2vzE0E7nx5a9qjGX0+GkNGjuKV0MLOZZPVjaKMAAA8oCCrPS0bB5+LTq29+o+Dzqxl8/JFvy7rL+Cz9uu2dnnzkuHTbMnTAvNRO56c3yN1zzxvvd83vrSl33+/c0jff7bk+G+tZte+drnDzJ0k0Tp4z63ykARBgBAnlGQlY9kC/7Lf7+Rz9cf1tPnIWeGpY+3HrmZz4dsWbOZSLdOS/m8ONG1cfzPc8L1Pg33rd05YpzPxyX2Vet37es+J2fc6ARZnho660bRVnwowgAASBH3k1WWzVdb1ue/9Fu3xvNPnLKtz8+fHja9vuWIULydtWu4by3ZFbJrYlnlz3HTEem3M247Xhna99/2Wtgk+9updIKsJMyuFR+KMAAAigDFGZZv39rnLbqH4u3wrcN9a+fsHgq5ZxINRh44LjQh2X/Trj7PXbjY59te+9bng24JnSCPu2eUz3e8Hs4Z/sVPDXsDKGnJQq2+Ii15LsVb41CEAQBQxDIVZ2Mu6lfoYaHIrNulvc9/23N9nweduLXPyeKsXetQ3H8wYYbPt7waZsv+/OhHPu868DWfn3p/Ug5GjFJTW+HF0sjGoQgDAKDEMGuGbK2xfDufk8VZshPkJftt4HOyUNto5Q4+T5uzMJz/3Gc+75fo/nj585/XeP0fZsxrzLBRxijaIhRhAACUidqKMwo11GX3Dbv4nCzU7vq/LX2+95iQt0oslfw+0f3xsdETa1x7r/+M8DlZpH3GXmqocBRhAABUEGbR0Bg9uoZZsZsSTUNu+sOmPh+//eqqrrmZz8ki7cjEXmon3Bu6PD749nc+T/s1NBsByg1/4wIAAF+cVakvl+sSITTMVqt38vnEHdfQ7YnGHpL06jk7qvcVwyVJx/XurjveGCdJWqZtS02Plzi+913Y72zgi2N93vWa0Hb/3Mc/9vnpD8I9abPn83OI0sRMGAAAaDBm1JCNNi2b+3xSnzV9fumsHXy+dP9wT9quPcIG2Ekvffajzxc/G+5J2+mq0IL/1kRTka9/mt3IEQPpoAgDAAA5Q3GGhuq3Qbgn7V8HhA2w3/hrH5/PTOyVtt1anX1ObkWdnIU75Na3fd73htA8ZMAzY3y+ZfjXPo/4aqrPcxeEtv5AvlCEAQCAvKNpCBoqOYv2h8Readcd2tPnIWeEDa736xk6O3ZcKvwsTZwemoc8+9Fkn6uWRkrS6Q9/6PP2Vw73eYdEPum+92qMcdCoCT6P+5kNsJE9ijAAAFAUkoUahRmy0bld2OD673uFzo7Dzg4t+G8/KjQSOXXntXw+eItVfF5nxdDKP2lOYlZs1PhpNZ6/8oUvfT7w5rABdrJgu/Hlr3xOboD9TWLJ5LipFHCVhiIMAAAUrWxm0JhNQ1027RZa6h+97Wo+n7P7uj4/eHzY1Pq1c0IB99Qft/X54n3DvWtVdl5veZ+XSszcJQu2u94c73NyA+yDE0smD7wlFHA7/zvc5/aPpz71+bHR3/s8ex4NSUodRRgAACgb3JOGpkr+rKyy7FI+77FRlxrnXnngxj6/ek7mDbAP22pVn5Ot/jsu1dLnDm3Ca85MFFjPf/KDz5c//4XPfRKF2nlPhM6RNyfuc0u2+x+dYRYPhUURBgAAyh73pCHfmiX2REtugH32buv4nNz0etjZoUPky38OBdygE8Os3Gl9w/LJndYNs25JL44JnSP/m7jPLdnu/8TE8sg9r3/D5+RM2yPvhvvbkH8UYQAAAGIWDcVhjeXD/WlHbROWT151UJh1SxZwZ+8WOkcm73PbrceKPnfr1NbnKTPn+5ycabtqSLi/rcofH3jf52tfCkXd7a+F7QDuGjHO508mzvCZvQTrxt8qAAAAdWjoRtbZ5rqe5xdY1CW5lPGwrbrp6qFRgXTO7utq0Kjo3rHLDthQQ8dMkSQ9cco22uKSYZKkO/tvoWPuHiVJOmOXtXTtS1HjkJ3WW16vfB4ah0jS29/+4vP9b4Xljbe+FrYDuPGVsASy/12jfN7hyrBk8tDbwv1vgz8Mm21XMoowAACAIpOPwo/CDpK08SodfT6i12q+CLvqwI19oVblon176IKno73Vjtymm+4bGRViB2y2sp54b6Ikae+NV9LguPX/Sh3baPKMeTVe86sfQyfIfw4Om23vOvA1n0+4d7TPydm1JUuSu8GVD5YjAgAAVACWW6KhfrfRSj6f3jcsezz/d+v5fOE+PXwefOp2Pic32/7PYWFvt826LePztDkLfX7vu+k+J2fX5i0qz82zKcIAAAAqGE1LkA/Jzba3WbOzz7cdtbnPDx2/lc//OmBDn/dINDYp15+98nxXAAAAyJnalkeyxBFNsfaK7X3etceKOu+JTyRJF++3wW+ahpQjijAAAAA0Sl33rlU/RsEGBBRhAAAAyLtkwUZBhkpHEQYAAIBUNbT7I0Ubyg1FGAAAAIoaRRvKDUUYAAAAygr7rKHYUYQBAAAA9ahe2FWhSENjUIQBAAAATZTN7FuyUKNRSWWjCAMAAABSUNtsWlOWT1LAlSaKMAAAAKBE0bSkNFGEAQAAABWivqJtzoJF6nHBkKyvQVHXOBRhAAAAACRlP7PWkPMp1GqiCAMAAACQN7UVapVcnFGEAQAAAEhdJRdn5pwr9BiKjpl1kDRjxowZ6tChQ6GHAwAAAKBAZs6cqY4dO0pSR+fczFxcs1kuLgIAAAAAyA5FGAAAAACkiCIMAAAAAFJEEQYAAAAAKaIIAwAAAIAUUYQBAAAAQIoowgAAAAAgRRRhAAAAAJAiijAAAAAASBFFGAAAAACkiCIMAAAAAFJEEQYAAAAAKaIIK2YLfpUGdIweC34t9GgAAAAA5ABFGAAAAACkiCIMAAAAAFJEEQYAAAAAKaIIAwAAAIAUUYSVigVzaNIBAAAAlAGKMAAAAABIEUVYMXMu5KljQ2ZWDAAAAChZFGFFLVGEPXdW4YYBAAAAIGcowoqZJb49P31WuHEAAAAAyBmKMAAAAABIEUVYqej5h5Bn/xgy94cBAAAAJYUirFTseF7IjxxeuHEAAAAAaBKKsFLRsk3I08YVbBgAAAAAmoYirBQts1rIMyYWbhwAAAAAGowirBQd+mDIT50YMveHAQAAAEWPIqwUtVsx5J8+L9w4AAAAADRYi0IPAHVotbQ0YEaUa5vZat5SWrwwvTEBAAAAaBJmwkrdbpeF/Nngwo0DAAAAQFYKXoSZ2Slm9q2ZzTOz0Wa2fR3n9jczl+HRJnFOCzO7JL7mXDP7xswuMLOCv9e82GD/kIecGzL3hwEAAABFqaDLEc3sEEnXSjpF0ghJJ0p63sx6OOe+q+XTZkpaN3nAOTcv8eFfJZ0k6WhJn0raQtJdkmZIui6nbyBN2SxNXDQ/vfEAAAAAaJRC3xN2lqT/OufuiD8+w8z6STpZ0nm1fI5zzv1QxzW3kfS0c+65+ONxZnaYomIsIzNrLal14lD7rEZfbDqvLf08NsoL59V9LgAAAICCKNgSPTNrJWlzSUOrPTVU0rZ1fGo7MxtvZt+b2bNmtmm159+Q1NfM1olfZxNJvSX9r45rnqdopqzq8X3276SI7H9ryMMGhMzSRAAAAKBoFHImbDlJzSVNqXZ8iqQutXzO55L6S/pYUgdJp0saYWabOOfiKSBdIamjpM/NbHH8Gn9zzj1Ux1j+JWlg4uP2KuZCrLalict0C/mTx9IdEwAAAICsFHo5oiS5ah9bhmPRic69Jektf6LZCEnvSTpV0mnx4UMkHSHpcEX3hPWUdK2ZTXLO3VPLdedL8jdUmVmj3ggAAAAA1KeQHQOnSlqsmrNeK6jm7FhGzrklkt6VtHbi8FWSLnfOPeyc+9g5d5+ka1T7PWblac2+Ic+dHjJLEwEAAICCKlgR5pxbIGm0pF2rPbWrpDezuYZFU1Y9JU1OHG4raUm1UxerCNrxp2qPq0L+39mFGwcAAACA3yj0csSBku4zs1GSRko6QVI3SbdIkpndK2mic+68+OMLFS1HHKvonrDTFBVhf0xcc7Ckv5nZd4qWI26qqAvjnWm8odTVdn9Ymw4hf/tqumMCAAAAUKuCFmHOuUfMrLOkCyStJOkTSb9zzo2PT+mm385qLSPpNkVLGGdIel/SDs65dxLnnCrpYkk3KVraOEnSrZIuyuNbKU0L5kiXdY3y+ZOigg4AAABAXhV6JkzOuZsUFUyZnutT7eMzJZ1Zz/VmSTojfkCSNjlc+vDBKP/0RWHHAgAAAFS4ghdhyKHalibu9LdQhA06Mv1xAQAAAPAqq1lFpWrROuS5vxRuHAAAAAAowirOcuuGPH1CyLSuBwAAAFJBEVZpDro35EFHFG4cAAAAQIXinrByVdv9YUt3DnnmxHTHBAAAAICZsIrWcdWQ588q3DgAAACACkIRVskOvi/kZxMd/bk/DAAAAMgbirBK1nGVkL99tXDjAAAAACoI94RVgtruDwMAAACQOmbCEOn1x5Anf1C4cQAAAABljiKs0lTNig2YIbVqG45vd3rIT50SMveHAQAAADlFEYaIJX4Ufv2xcOMAAAAAyhxFGGpaatmQ3ZLCjQMAAAAoQxRhqGnv/4T84j9CZmkiAAAA0GR0R6xktXVN7NYr5I8eSXdMAAAAQJljJgwAAAAAUkQRhrrtclHIY54OmaWJAAAAQKNQhCFSW+v6noeHPPT89McFAAAAlBmKMGRv0fxCjwAAAAAoeRRhyF6nNUOeNzNkliYCAAAAWaMIQ021LU3c75aQH+uf+rAAAACAckARhux1Wj3kHz4q3DgAAACAEkYRhsZps0zIC+eGzNJEAAAAoE4UYahbbUsTD7435GdOTX9cAAAAQImiCEPjrNAj5G+HZz6HWTEAAACgBoowNJ01D9m5wo0DAAAAKAEUYchebUsT97gi5FevqPl5AAAAADyKMDRdj/1CHnVH4cYBAAAAlACKMORPcmniZV25NwwAAAAQRRhyrc95IQ85r/bzAAAAgApFEYbGqe3+sC2ODfmTx2p+Hh0TAQAAUOEowpA/LdoUegQAAABA0aEIQ/4cdG/95wAAAAAVpkWhB4AyULU0UfrtEsOVNyvMeAAAAIAixkwY0vXpkyFzfxgAAAAqEEUY0vX8OYUeAQAAAFBQLEdEbtW2NNFzGY4BAAAAlYOZMKRrk8ND/nJIyCxNBAAAQIWgCEO6dhkQ8gssTQQAAEDloQhD/lQtTTx/UjhmiR+52ma8mBUDAABAGaMIQ+EstWyhRwAAAACkjiIMhbPnwJDfv79w4wAAAABSRBGGwum+fcjDBmQ+h6WJAAAAKDO0qEf+1du2HgAAAKgczIShOGx9csivXlm4cQAAAAB5RhGG4tD7rJDfvS3zOSxNBAAAQBlgOSLSVdvSRLOQW7SWFs2P8i/fpDc2AAAAIAXMhKH4HPpIyA/8vnDjAAAAAPKAIgzFp8uGIc+fFbJzIbM0EQAAACWK5YgonGy6Jm50sPTxoCi/MTDzOQAAAEAJYSYMxW23S0N+++bCjQMAAADIEYowFLdkw47asDQRAAAAJYQiDKUj2cZ+9F2FGwcAAADQBBRhKA5V94cNmCG1apv5nF6nhPzKpZnPAQAAAIocRRjKC0sTAQAAUOQowlB8spkV27x/yGOeSmVYAAAAQC5QhKE09flbyM+fU7hxAAAAAA3EPmEobrXtJZbsmuiWZP7cBXOky7pG+fxJ0bUAAACAAmMmDKWvx74hf/pk4cYBAAAAZIEiDKVv9ytCfv4vmc+hYQcAAACKBMsRUTpqW5rYrJYf4yWL8z8mAAAAoIGYCUN56XthyLXNigEAAAAFRBGG8rLpkSF/9kzmc1iaCAAAgAJiOSJKU21LE5Oat5QWL4zyvBnpjAsAAACoBzNhKF/73x7yffsVbhwAAABAAkUYylf33iHPmJD5HJYmAgAAIGUsR0Tpy2Zp4pp9pa+HRfmDB9IZFwAAAJABM2GoDPvdEvJLF9Z+HgAAAJBnFGGoDGb1n8PSRAAAAKSA5YgoL9ksTdzkcOnDB6M84Z10xgUAAADEmAlD5dllQMjP/LFQowAAAECFoghD5bHEj/3caZnPYWkiAAAA8oTliChf2SxNbLeiNHtKlOfPSmdcAAAAqGjMhKGyJbsmPnxY5nOYFQMAAEAOMROGylDbrFiXjUL+6fN0xwQAAICKxEwYUKXTmiFP+bRw4wAAAEBZowgDqhw+KORHj858DksTAQAA0EQsR0TlqW1pYpuOIc+bnu6YAAAAUDGYCQMyWalnyONHZD6HWTEAAAA0AkUYkMmBd4X8+LGFGwcAAADKDkUYkEnr9iEvWRSyW5L5fGbFAAAAkCWKMFS2qvvDBsyQWrXNfM5WJ4T85InpjAsAAABliyIMqM8O54T8zSuFGwcAAADKAkUY0BAdVg553BuZz7msK8sSAQAAUCuKMKBKNksTj3gyZBp2AAAAoBEowoCGaNspZLc45GTzjio06wAAAEAGFGFAJtnMim17WsjMigEAACBLFGFAYyWLsNo2dAYAAACqoQgDcqHjqnU/z9JEAAAAxCjCgPpk1bDjiZrHFszJ77gAAABQkijCgFxYatmax57gPjEAAADU1KLQAwBKStWsmFT/ssLv3818fMGcaC8xSTp/UnRNAAAAVAxmwoB8abNMyHOnFW4cAAAAKCoUYUC+HHxfyA8dUrhxAAAAoKg0eTmimXWQtLOkL5xznzV9SECJqG9p4grrh/zLN+mMCQAAAEWvwTNhZjbIzP4U56UkjZI0SNJHZvb7HI8PKA/LrRNy8l4xWtcDAABUnMYsR9xB0utx3l+SSVpG0mmS/p6jcQHl5dCHQ37smMKNAwAAAAXXmCKso6Rf4ry7pMedc3MkPSdp7VwNDCgpVUsTz5+U+fk2HUJeNDfzOcyKAQAAVITGFGETJG1jZksrKsKGxseXlTQvVwMDytaafUP+6sXCjQMAAAAF0Zgi7FpJD0j6XtIkScPj4ztI+jg3wwJKVNWM2IAZUqu2mc/Z54aQnzk18znMigEAAJStBndHdM7dZGbvSFpV0ovOuSXxU9+Ie8KA+jVvGfKSRYUbBwAAAAqiUfuEOedGOeeedM7NNrPmZtZT0pvOuRE5Hh9Q3jY8MOR3bivcOAAAAJCaxrSov9bMjo1zc0mvSnpP0gQz65Pb4QElLJulif0uC/m1KzOfw9JEAACAstKYmbADJX0Y570lrS5pPUX3il2ao3EBlcESfwSteci//pz5fAoyAACAkteYImw5ST/E+XeSHnXOfSnpv5I2ytXAgLKSzazY7/8b8v37pzMuAAAApK4xRdgUST3ipYi7S3opPt5W0uJcDQyoON17hzyrlv3GkpgVAwAAKEmNKcLukjRI0ieSnKSqjY62lvR5jsYFVLbVtgv57VsLNw4AAADkXGNa1A8ws08Utah/1Dk3P35qsaTLczk4oCxVLU2Uap/BOuAO6Zr1o/z6VemMCwAAAKlocBEmSc65xzIcu6fpwwEg6bd7iVlzycUrfX/5NvP5C+ZIl3WN8vmTokIPAAAARalR+4SZ2Y5mNtjMvjKzsWb2jJltn+vBAWUvq4Ydd4R8z57pjAsAAAB505h9wo5Q1IxjjqTrJd0gaa6kYWZ2eG6HB0DdE/++sXhByD99mf5YAAAA0GSNmQn7m6RznHOHOOeud85d55w7RNK5kv6R2+EB+I3fXR3yA7W0sadrIgAAQFFrTBG2hqTBGY4/o2jjZgCNkc3SxB77hrxofsgL5mQ+n4IMAACg6DSmCJsgqW+G433j5wCkYefExPOgIws3DgAAADRIY7ojXi3pejPrKelNRXuF9ZbUX9LpuRsaUMGyaWO/2dHSyxdH+YcP0xkXAAAAmqwx+4TdbGY/SDpb0sHx4c8kHeKcezqXgwOQpQ4rSzMnRvmHTzKfQxt7AACAotCoFvXOuSedc72dc53jR29J/zOzbjkeH4BsHD4o5Ef+ULhxAAAAoF6N2qy5Fj0kvSepeQ6vCSCbpYntVgx5IQ04AAAAilmjZsIAFLH19g75xVp2jaBrIgAAQMHkciYMQL5lMyu259XS5/EuEh8+lM64AAAAkDVmwoByY4k/1kstG/LE0ZnPZ1YMAAAgVVnPhJnZxvWcsm4TxwKgIZK9BdIAACAASURBVLKZFTvyaem2HaLMXmIAAABFoSHLET9QtCeYZXiu6rjLxaAA5EiHriEvXhCy448qAABAoTSkCFs9b6MAkH+bHS29d0+UX7ow8znsJQYAAJB3WRdhzrnx+RwIgCbIZmnizv8IRdiHD6YzLgAAANRAYw6gErVoE/LsKZnPoWEHAABAXlCEAeWmalZswAypVdvM5xzyQMgPHJjOuAAAACCJIgyoTCttEvKsyYUbBwAAQAWiCAMq3cpbhDz2xcznsDQRAAAgZyjCgHKWzdLEg+4J+Zk/pjMuAACACtbgIszM3jez9zI8RpvZCDO7x8x2asD1TjGzb81sXnyN7es4t7+ZuQyPNtXOW9nM7jezn81sjpl9YGabN/S9AhWhReuQ3ZLMGQAAADnTmJmwFyStIelXSa9IGi5ptqQ1Jb0raSVJL5nZvvVdyMwOkXStpEslbSrpdUnPm1m3Oj5tZvwa/uGcm5e45rKSRkhaKGkPST0knS1pekPeJFCRNj8m5MePzXzOZV1ZlggAANAEDdmsucpykq52zl2cPGhmf5e0mnNuNzP7p6R/SHq6nmudJem/zrk74o/PMLN+kk6WdF4tn+Occz/Ucc2/SprgnPu/xLFx9YwDKH/Z7CXW5zxp9J1RHvd6OuMCAACoMI2ZCTtY0kMZjj8cP6f4+XXruoiZtZK0uaSh1Z4aKmnbOj61nZmNN7PvzexZM9u02vP7SBplZo+a2Y/x8snj6xlLazPrUPWQ1L6u84GyZRZypzVDHv9mzXNp1gEAANAojSnC5ilzkbRt/FzVdefXc53lJDWXVH2n2CmSutTyOZ9L6q+o0Dosfr0RZrZ24pw1FM2kjZXUT9Itkq43s6PqGMt5kmYkHt/XM3agtGXTsOOIJ0J+4rh0xgUAAFABGrMc8T+SbokbXbwryUnaStJxki6Lz+kn6f0sr+eqfWwZjkUnOveWpLf8iWYjJL0n6VRJp8WHm0ka5Zw7P/74fTPbQFFhdm8tY/iXpIGJj9uLQgyVrtXSIS9eUPe5C+ZE94pJ0vmTfvu5AAAA+I0GF2HOuUvM7FtJf5J0ZHz4C0nHO+cejD++RdLN9VxqqqTFqjnrtYJqzo7VNpYlZvaupORM2GRJY6qd+pmk39dxnflKzNxZckkWAKnHvtKY+m7xjFGQAQAA1KkxM2Fyzj0g6YE6np+bxTUWmNloSbtKejLx1K6qv6GHJMmiaqmnpI8Th0eo5v1o60gan801gYqTTcOOPa7KvggDAABAnRpVhElSvBxxfUVLB8c457Jdfpg0UNJ9ZjZK0khJJ0jqpmgmTWZ2r6SJzrnz4o8vVLQccaykDoqWIPaUlNxh9pr/b+/OwyWrynsB/xajKAg4AKKiXmNwiog4RkWQIAqKokZBUcHxGk0ixkTpGCXRYHJNjHOioCA4QSSgIiqDGCPiBE4IIhIgYjPI1CBTN937/lF1sorTdU5VdZ8azqn3fZ56+le71q5apfup7o9v77WTfKeUsizJ8WmdKvna9gNYF6XL5aPfeu/o5wEAsAQMXISVUrZJayXE3dK691ZJsmUp5cwk+zdN89t+36tpmuNKKfdM8o607vl1XpK9m6aZ6VrtkKTzjrFbJfl4WqcwrkjrurNdm6b5fsd7/qCUsl9a13m9I8klSd7U7t4BC+X7H+s9xqmJAABrWdeFOe6e5BFN01yQJKWUhyf5VJIPprVqYd+apvloko/O8dpus54fkuSQPt7z5CQnDzIPIP2dmjhjg42TNata+Xd9XcYJAEDWbYn6ZyZ5/UwBliRN05yf1imBz1qoiQET7oWfrPmoZ/Ye775iAABJ1q0TtkGSVV22r8q6FXXAJOrVFdvhSTXfflPNN/x6uPMCAFjk1qVo+kaSD5RStp/ZUEq5b1oLYpyxUBMDFpFd31rzp/bpPV5XDACYYutShL0xrZsZX1pKubiU8qu0Fr/YIvWGycBSMtMVW7a8++uPf03Nq26p+aYrhjsvAIBFaF1u1vzrJI8ppeyZ5KFprY54ftM0py/05IBFaI/DkjMOa+VPz3mP9MoKigDAlFnna7iapjmtaZoPNU3zwaZpTi+l3L+U8sneewJL2s4H1nzz1eObBwDAhFrIhTTukeQVC/h+wKSZOS3xsBXJJnftPf6BT6n5lLf0Hu9aMQBgCljNEBie5x9Z8/kn1bzmjtHPBQBgQijCgOHZoOOy0+13rvm4l45+LgAAE0IRBqybQU9NPOD4mn9zTu/xTk0EAJaovldHLKX8R48hW63nXIClrJSat31kctV5rdzPtWIAAEvIIEvUr+jj9WPWYy7AYjXTFUv661odcFzy/ke0cue1YnOxjD0AsIT0XYQ1TXPwMCcCTJGNNq35Xjsm11zYyqf85XjmAwAwQq4JAxbWoNeKvayjE3b+ib3Hu1YMAFjkFGHAeG24cc2bb1vzL07uva+CDABYhBRhwPAM2hV7+ZdqPvlNw5sXAMAYKcKAyXHXe3bffuv1vffVFQMAFglFGDAag3bF9nlfzcc8d3jzAgAYMUUYMJketm/NNy2vec3q0c8FAGABKcKAyfeQvWr+zAt7j3dqIgAwwRRhwOgNemrivh+u+aqf1bx65cLPDQBgyBRhwOQrpeYd96n58wf03ldXDACYMBuNewLAlJvpiiX9FUnP+UBy4Vda+YqfDG9eAABDohMGLF7b7VTzqX89vnkAAAxAEQZMjkGvFTvgczX/9Lje452aCABMAKcjApOpn9MUN9yk5i3uk9x0RSv/9zeHOjUAgPWhEwYsDa/4Ss0nvq73eF0xAGBMdMKAyddPV+wud6+56bih8+pVw5sXAMA60AkDlp5dDqr52Of1Hq8rBgCMkCIMWHp261gp8ZoLa9YVAwAmgNMRgcWln1MTO2/u/PD9kvNPbOV+bu4MADBkOmHA0rb3e2u+4se9xx++vdMSAYChUoQB0+M+j675G+8a3zwAgKmmCAMWr0Fv7rx/x82dz/3U/GMt1gEADIkiDJgeG25c82Zb1/yLr6w9FgBgSBRhwNIwaFfs5V+q+eQ/H968AABmUYQB02mL+3Q8KXMOS+LURABgQSnCgKVn0K7YHx899CkBAMxQhAE84Mlrb/ufs0c/DwBgKrhZM7C09XNz526Of1n37Stvad1LLEmWLW+9PwDAAHTCAHq59NvjngEAsIQowoDpMci1YgccV/MJrxzuvACAqaIIA+jmvrvU3KypeeUtd85WTQQABqQIA6bTIF2xPd5Z82deMNx5AQBLniIMoJedOxbpuPai7mN0xQCAPinCAGa6YsuW9x57v8fX/I13D29OAMCSpQgDGMSLjqn53KO7j9EVAwDm4T5hADP6uafYBh0/m5tukdx+Uyv/z3eHOzcAYMnQCQNYVy87qeYvHNR9jK4YADCLThhAN/10xbZ6QM1r7qi5cxl7AIBZdMIAFsKub6356L27j9EVAwCiCAPorZ97ij3+NTXfeHnNN1873LkBAIuOIgxgoe1ycM1H7dV9jK4YAEwtRRjAIPrpiu3+1zXfdkPNv7t6uHMDABYFRRjAMHVeK3bMc7qP0RUDgKmiCANYV4NeK3ZLx/VhnaspAgBTRREGMCqPfGHNn3tx9zG6YgCw5LlPGMBC6Oe+Ys/8h+S8L7TyFT+p25tmuHMDACaKThjAODzwqTWf/Ofdx+iKAcCSpAgDWGj9XCv2gk/WfOEpo5kXADARFGEA41BKzXe7d83nn9R9vK4YACwZijCAYeqnK/ayjsLrlLeMZl4AwNgowgDGbfNtay4b1nzFT7uP1xUDgEVNEQYwKv10xQ74fM2fn2MZewBgUVOEAUyS7XeuefWqmq+5qPt4XTEAWHQUYQDj0E9XbLdlNX9qn9HMCwAYOkUYwKR67CtrbtbU/N2Pdh+vKwYAi4IiDGAxeOl/1Pzt941vHgDAetto3BMAmHozpyYmc3ew7vOomu927+Tm37byF9/QffzKW5LDt2/lZctbnwEATASdMIDF5pWn1nzR12tumtHPBQAYmE4YwCTppyu26RY13/exyW9+2Mpfe1v38bpiADBRdMIAFrP9P1fzz08Y3zwAgL7phAFMqn66YqXUfJetkttuaOWfn9h9vI4YAIydThjAUnFgR+H11b8c3zwAgHnphAEsBv10xba6f80bbJysWdXKV1+w9ljXiQHA2OiEASw2MwXZYSuSTe7afcyBHdeHffZFo5kXANAXRRjAUrTNw2u+49b5x668JTlsy9Zjri4bALBgFGEAi1k/XbFdDl5727UXD3deAMCcFGEAS93uf732tk/t032srhgADJ0iDGAarbmj5gu+PL55AMAUUoQBLBX9nJo44wWfqPkrh3QfoysGAEOhCAOYRg96Ws0bbFzzhaeMfi4AMGUUYQBL0SBdsZedVPOX/6z7GF0xAFgwijCAaXfvHTuelBqv+vnIpwIA00ARBrDUzXTFli3vPfZFx9T82Rd2H6MrBgDrZaNxTwCAEZkpxpK5i6cdnlTz6lU1r/jN8OYFAFNGJwyA7vY4rOajntF9jK4YAAxMEQYwjfpZuGPnA2u+4/aaL/lW9/EKMgDoiyIMYNr1U5A9+wM1n/DK0cwLAJYoRRgAvT10n44nHSso/viz3cfrigHAnBRhAAzmJcfVfPo7am6a0c8FABYhqyMCUPWzguL2j6l5s62TW69v5c/v3338yluSw7dv5WXLW58BAFNMJwyAdffqb9T8m3NqvvWG0c8FABYJnTAAuuunK7bpFjU/8oXJeV9o5WOf2328rhgAKMIA6EM/Bdkz/6EWYTd23Ny5WTPcuQHAIuN0RAAW3kP2qvkLB3cfYwVFAKaUIgyAwfRzX7F9P1zzZWfVvGb1cOcGAIuAIgyAhVc67iW23U41H/Oc7uN1xQCYIoowANZdP12xzvuKXfPLmq+/tPt4BRkAS5wiDICFMVdBtkHHGlCPf23Nn3r26OYGABNEEQbA6Oz6VzXfcVvNV/yk+3hdMQCWIEUYAOPxjHfX/JkX9B6vIANgiVCEAbDw+rlW7FH7d99+ztFDmxYATAJFGADjd8DxNZ/57rnHzdAVA2AR26j3EABYDzNdsWTugum+j6n5Llslt93Qyt9413DnBgBjoBMGwOj0c5riq06v+dxP1dw03cfrigGwyCjCAJgsm21V8+bb1vz5Oa4hA4BFRhEGwHj00xU76Ks1/+acmldc3n384dvriAEw8RRhAEyuu9y95se8ouajnjX6uQDAAlGEATB+/XTFnv43Nd9xa82X/tfaY10nBsAEU4QBsPg84+9r/sLB849VkAEwYRRhAEyWvm70/OKaN9h4NPMCgAWiCANgcTv4q73HzNAVA2ACuFkzAJOrnxs9b/3Atbd9fdnQpgQA60snDICl52fH1+wmzwBMGJ0wABaHfrpiM+75kOTai1r5uJcMd14AMCCdMACWnpd/qebLf1DzqlvXHpvoigEwUjphACw+vbpiG3asmPjgPZKLz2hlN3kGYALohAGwuM0UZMuWd399v4/VfOPlNd9wWffxumIADJkiDIDp8dhX1/zJvcY3DwCmmiIMgOmx29tqXnNHzT/9fPfxumIADIEiDIClYea0xMNWJJvctff4Fx1b86lv7z1eQQbAAlGEATCddnhSzRt0LOTxo0+Pfi4ATBVFGABLz6BdsQP/o+YzDqvZjZ4BGAJFGABs87Ca73bvmk987ejnAsCSpwgDYGkbtCv2qjNq/u8ze4/XFQNgQIowAOjUWajd+6E1f/FPRj8XAJYkRRgA02PQrthLT6j5olNrXr2q+3hdMQD6sNG4JwAAYzFTkCVzF0wbbVrz9o9Jlp/byp969nDnBsCSphMGAP04oOOGztddXPP1l3YfrysGwBwUYQDQj9LxV+Yur6z5qGeNfi4ALGoTUYSVUv6klHJJKeW2Uso5pZSnzjP2oFJK0+VxlznGH9p+/f3D+wYALGqDXiu2+7Ka13RcH/azf+8+XlcMgA5jL8JKKS9O8v4kf59k5yT/leSrpZQd5tntxiT36Xw0TXNbl/d+XJLXJvnpQs8bgCVq0ILs+UfW/PVDhzcvAJaMsRdhSd6c5BNN0xzZNM0FTdO8Kcmvk7x+nn2apmmu7HzMHlBK2TzJZ5K8Jsn1802glLJpKeXuM48kW6z71wFgqvyf3WreZPOa//P/dR+vKwYw9cZahJVSNkmyS5JTZ710apI/nGfXzUspl5VSLi+lnFxK2bnLmI8k+UrTNKf3MZVDk6zoeFzexz4ALHUD3+i546+cH3x8ePMCYFEbdyfsXkk2THLVrO1XJdlujn1+keSgJPsmOSDJbUnOKqU8ZGZAKWX/tIq7fs8LeU+SLTse9+tzPwCo7navmu96z5rP/kj38bpiAFNpUu4T1sx6Xrpsaw1smu8m+e7/DizlrCTnJvnTJH9WSrl/kg8keUa368TmeM/bk9ze8Z4DTR6AKdDPfcU6HXRK8tEntPJZ/9J7/MpbksO3b+Vly1ufB8CSNO4i7Jokq7N212ubrN0d66ppmjWllB8kmemE7dLe/5yOYmrDJLuWUt6YZNOmaVav78QBmGL9FGSdnbBN757cfmMrn3PUcOcGwMQb6+mITdOsTHJOkj1nvbRnku/08x6lVWk9OskV7U1nJPmD9raZxw/TWqTj0QowAEbuoFNqPvPvxzcPACbCuDthSfK+JMeWUn6Y5Oy0lpTfIcm/JUkp5Zgkv2ma5tD283emdTriRUnunuTP0iq03pAkTdPclOS8zg8opdyc5Nqmae60HQDWWz9dsS06TvjY+K7Jqlta+Yy/6z7eqYkAS9rYi7CmaY4rpdwzyTvSuufXeUn2bprmsvaQHZKs6dhlqyQfT+sUxhVJfpRk16Zpvj+6WQPAOnrFl5Mj92jlHx1Tt99+03jmA8DIjb0IS5KmaT6a5KNzvLbbrOeHJDlkwPffrecgAFhf/XTFtnpAzds9Krnyp6185NO7j9cVA1hyxr1EPQAsTf3cY+ylJ9R86/U1X3dJ9/GWtAdYEhRhADAunbdE2evwmo993ujnAsDIKMIAYNj66Yr9wYtqXtXR5ZrrWrHDt9cRA1ikFGEAMGme+Iaaj37W+OYBwFAowgBglPrpij2lY/2pm66secVv1h7rOjGARUcRBgDj0k9BtstBNR/1jPnfT0EGsCgowgBgku3+9prvuH188wBgwSjCAGAS9NMVe+6/rr3t7A93H6srBjCxFGEAsFg8ZM+1t531/t77KcgAJspG454AADDLTFcs6V00bXy3uqT9jz8z3HkBsCB0wgBgMXv5F2s+/Z01N0338bpiAGOnEwYAk6xXV2zrB9a82T2SW69r5WP3HfrUAFg3OmEAsFS8+oyar76g5h99uvt4XTGAsVCEAcBiMdMVW7a8++ubblHzE99Q8xmH1XzztUOZGgD9U4QBwGLTz3L2Tzmk5ns8uOaj9uo+XlcMYGQUYQCw1B38tZpvu6HmFZePfi4AKMIAYFHrpytWSs1P/Yuaj967+3hdMYChUoQBwDR5wutrXnVLzZ0LeQAwVIowAFgq+umKddrzXTUf+7zuY3TFABacIgwAptVOB9TcrK75/JO6j1eQASwIRRgALEWDdsX2O6LmU94yvHkBkI3GPQEAYMhmCrJk7g7Wg3e/8/iZcV97W/fxK29JDt++lZctb+0DQF90wgCAO3vVGTWf94Wa16xeeywAA9MJA4Bp0k9X7G73qvmev5dc+6tWnm9Je10xgL7phAHAtOrnurGXf6nm6y6u+eIzhzs3gCVMEQYAzG3DTWp+8iE1n/ia0c8FYIlQhAEA/XXFnvSGmjuLs6+8uWbL2AP0pAgDAO6sn4LslafVfEHHKYudi3coyAC6UoQBAIPb8r417/Ckmv/DaYoAvSjCAIC59dMV++Njar70W93H6IoB/C9FGACwfkqpecv71/zFN6w9NlGQAVNPEQYA9KefrtiBJ9Z80ddrvvX64c4NYBFxs2YAYHBz3fR5s61qfshetRD75DO6v48bPQNTSCcMAFg/c3XInvuRmjs7YTdcNrq5AUwgRRgAMHxP6biX2NF7dx/jWjFgSijCAIDhe+Kf1HzH7TVffEb38YdvrxgDlixFGACwcPpZvONZ7635xNeNZl4AE0QRBgAMx1wF2SP2q3nDjWv++qFrv4dTFIElSBEGAIzPwafW/LN/H988AEbIEvUAwPDNtaT9Vh03d95up+TKn9x5v2ZNzZazB5YInTAAYDK8tEsn7LiXjn4eAEOmEwYAjNZcXbHS5b8NX/6DmjvH6ooBi5hOGAAwuR68R81HPn188wBYQIowAGB8ei1pv9/Har7l2pp//b2araAILDKKMABgMswUZMuWd399z3fVfPzLRzMngCFQhAEAi8NOB9TcrK755t/WrCsGLAKKMABgsvQ6RTG5c1fsyD/qPkZBBkwoRRgAsPh0dsVWdRRYF585+rkADEgRBgBMrn66Ynv/c80nvqb7GF0xYIIowgCAxe3hz625bFjzDz8x+rkA9EERBgAsDv10xV52Ys3ffE/NTVOzrhgwZoowAGDp2ObhNd/t3jV/7kXdxyvIgDFQhAEAi08/XbFXnVbz8h/VfPX5w50bQA+KMABgadpk85p3eknNx+zbfbyuGDAiijAAYHHr675if9d9+2nv6L5dQQYMkSIMAFg6+lq844s1/+SzNd9x+3DnBtCmCAMApsu2j6h5+51rPvZ53cfrigELTBEGACxN/XTFDji+5msvqvmmK4c7N2CqKcIAgOlVSs077l3zEbt1H68rBiyAjcY9AQCAoZvpiiVzF0/P+WBy4SmtvOaOuv2Ut3Qfv/KW5PDtW3nZ8tZnAPRBJwwAmC79nKZ44Ek1n9+RV94y3LkBU0EnDACYXnN1yLZ7ZM33e3xy+fdb+ehndX8fXTFgADphAADzefFnar7xNzVf+6vu4103BvSgCAMASOY+TbFz8Y7Hvbbmo/fp/Z4KMqALRRgAQL+e9lc1N6tr/s9/HP1cgEVLEQYAMFs/i3fs/7maf3BEzbet6D5eVwxoU4QBAKyL+z2u5m0fUfNc9xjrdPj2ijGYYoowAID5DLqk/e031Xz6YUOdGrA4KcIAANZX5+Idz/5AzT/+dM3XX7r2fk5RhKmkCAMA6Fc/XbGHdqya+OCn13zUM+d/bwUZTA1FGADAuuinINvv4zWvuaP/91aQwZKmCAMAGIUXf3btbRef2Xs/BRksORuNewIAAIveTFcsmbtQuv/j19524muGNydgYumEAQCMywYb13zyIb3H64rBkqAIAwBYSP1cKzbjoFNq/sWXa/7l13p/joIMFi2nIwIADEuv0xTv8aCa/+CPk5/9eyt/6Y11e7Om9+esvKV1A+gkWba89bnAxNIJAwCYBHu9p+aNO4qoT+w5+rkAQ6UIAwAYhZmu2LLlvce+pmPVxBsuq/nMw3vv6zRFmHiKMACAUernmrG73qPmPd9d8zmfrPnma3p/loIMJpIiDABgku20f8332anmI3Yb7H0UZDAxFGEAAOMyyEqKSfKSL9R8x201f+1tCz83YGgUYQAAi0UpNb/gEzWf11Gc/c93e7+PrhiMlSXqAQAmQa/l7Gd70NNqfuizk1+c3MrHHzjY51reHkZOEQYAMGkGLcie/f5ahG2wcbJmVSsf99LhzA9YL05HBABYSl51Ws2//l7NF3y5975OU4SRUIQBAEyyQRfv2PJ+Ne9yUM1fOaTmVR2LesxFQQZDowgDAFgsBi3Idn97zZttXfPHdx3scxVksKAUYQAA0+C136r51utqPvPw0c8FppwiDABgMRq0K7bxZjXv876az/lkzTdf0/t9dMVgvSnCAAAWu0ELsoftW/P9Hl/zkXss/NyAtSjCAACm2f6frXlVR2frJ5/rva+uGKwTRRgAwFIyaFes097/XPNpf1Pzmjt676sgg74pwgAAlqpBC7KHP7fmu96r5qP3Xvi5wRRThAEAsLbXnFnzdf9d86+/33vfw7fXEYN5KMIAAKbB+qym+OSOGz0f95KFnxtMGUUYAMC0GbQge9Ibai4d/3z81nvn3891YtCVIgwAgP69/Ms1f/9jNSuyoG8bjXsCAACM0UxXLOmvkLr3jjVv/cDk+ktb+WNPnX+/lbe0rhVLkmXLW58LU0onDACAlkFPUzzolJpvv3F484IlRhEGAMC62XCTmp/7r2u//o13d9/PtWJMOacjAgCwtkFPU3zInmtvO/foBZ0SLBWKMAAA5jdoQTZjs62TW69v5bM/0n2Ma8WYQk5HBABgOF7xlZrP+pea19zRfbzTFJkSOmEAAPRvkK7Y5tvUvOX9kxW/buXP7T+cucEioRMGAMDwvfLrNV/x45rXrO4+XleMJUwnDACAdTNIV6xzJcUHPDm57KxWPnrv4cwNJphOGAAA62+mIFu2vPfYFx5d83UX13zJt7qP1xVjiVGEAQCwcPq54XMpNT/pT2s+4ZW9319BxhKgCAMAYHye/Oc1d56yeMKrRz8XGBFFGAAAw9FPV6zTq8+o+ZJv1nzD/3QfryvGImVhDgAAhq+fRTy2uE/ND9s3ueBLrXzUXsOdG4yYThgAAJNnn/fVvHpVzeef1H28rhiLiCIMAIDRGvQ0xf2OqPmUt/QeryBjwinCAACYbA/eveaNNq35jL8d/VxgASjCAAAYn0G7Ygd9reYfHVvzqtu6j9cVYwJZmAMAgMnQz+IdW92/5m0enlx9fit/8o96v//KW5LDt2/lZctbnwdjoBMGAMDi9LKORTpuurLmS7/de18dMsZIEQYAwOTp5zTF0vFP2V3/quYvHDTUqcH6UoQBADDZ+inIHv/amjfouOLmW+/t/f66YoyYIgwAgKXl4I7FO77/scH2VZAxAhbmAABg8ehn8Y6tH1jz5tsmv7uqlf/9FUOdGvRLJwwAgKWrsyt22Vk1/+TzvffVFWNIdMIAAFic+umKbbpFNz6G9QAAFLFJREFUzQ/bN7ngS6182tvr9usvHcr0YC46YQAATId93lfzdjvVfNSzeu+rK8YCUoQBALD49bOCYqcDT6h5zaqaf3Rs730P314xxnpRhAEAsLQMWpC94BM1n/G3w5sXtCnCAACYbg96Ws2bbV3zCa+efz+nKLKOFGEAACxdg3bFXnV6zZd8s+abfzv/fgoyBqAIAwCAGXfZsuYHP73mI/cY/VxYshRhAABMh0G7Yvt9vOZVt/T/Obpi9DARRVgp5U9KKZeUUm4rpZxTSnnqPGMPKqU0XR536RhzaCnlB6WUm0opV5dSTiql7DiabwMAwMQbtCDb9yNrb/v5ib33U5DRxdiLsFLKi5O8P8nfJ9k5yX8l+WopZYd5drsxyX06H03T3Nbx+tOSfCTJE5PsmdZNqU8tpdxt4b8BAABL3u/vtfa2r/5lzU0zurmw6G007gkkeXOSTzRNc2T7+ZtKKXsleX2SQ+fYp2ma5sq53rBpmmd2Pi+lHJzk6iS7JPnW+k8ZAIAlY6YrlgzWrdp0i+T2m1r5+AN7j195S+seY0mybHnrc5lKY+2ElVI2SaswOnXWS6cm+cN5dt28lHJZKeXyUsrJpZSde3zUzBWW180xj01LKXefeSTZop/5AwCwxAxymuKrz6z519+r+bcX9v4cpylOtXGfjnivJBsmuWrW9quSbDfHPr9IclCSfZMckOS2JGeVUh7SbXAppSR5X5JvN01z3hzveWiSFR2Py/v/CgAATKXNtqr5ES+o+VP7jH4uLCqTcDpiksw+ibZ02dYa2DTfTfLd/x1YyllJzk3yp0n+rMsuH07yqCRPmefz35NWoTZjiyjEAACm2yCnKT7rH5Ofn9B+0vFP2ZP+b+/PcZri1Bl3J+yaJKuzdtdrm6zdHeuqaZo1SX6QZK1OWCnlQ2l1zHZvmmbOoqppmtubprlx5pHkpj7nDwAAd/bKjittftVx82enHdI21iKsaZqVSc5JawXDTnsm+U4/79E+3fDRSa7o3FZK+XCS5yd5etM0lyzMjAEAmEozXbFly3uPvceDan7QbjUf3cdpiq4VmwqTcDri+5IcW0r5YZKzk7w2yQ5J/i1JSinHJPlN0zSHtp+/M63TES9Kcve0TkF8dJI3dLznR5K8JMlzk9xUSpnptK1omubWoX8jAACWpkFXUnzBkck//V4r39hxYtaNfRRzLFljL8KapjmulHLPJO9I655f5yXZu2may9pDdkiypmOXrZJ8PK1TGFck+VGSXZum+X7HmNe3//zmrI87OMnRCzl/AADoy04vSX7y2VY+Yvfe410rtmSNvQhLkqZpPprko3O8ttus54ckOaTH+5UFmxwAAHQzaFdsz7+rRVizum4/4dW991WQLSnjXpgDAACmzyu+UvMl36z55mt77+u6sUVvIjphAACwqA3aFbv3jjXvuE9yYbsoO2qvhZ8bE0cRBgAAC2nQguw5H6hF2G031O3X/XfvfZ2muCg5HREAACbFU/+i5qP3HmxfpykuGoowAAAYlpmu2GErkk3u2nv8E15f85o7aj7rAws/N8ZGEQYAAJPo+Z+o+ewP1Xzbit776opNNNeEAQDAKAx6rdj/eVrN2z4iuernrXzkHgs/N0ZKJwwAAEZt0NMUDzyp5s7FO350bO99dcUmjiIMAADGqZ+CrJSa9zis5jP+tuZrf9X7sxRkE0ERBgAAi8nOB9a81QNqPuqZNTfN6ObDwBRhAAAwKQY9TfFVp9VcNqz5+APXHjubrtjYKMIAAGCxKh3/nH/dt2r+9fdqvv7S3u+jIBspqyMCAMAkGnQ1xc23rXnHvZMLT2nlTz5j4efGetEJAwCApeY5H6y5WVPz6e/sve/h2+uIDZkiDAAAJt2g14p1evmXav7xZ2q++dr593OK4tAowgAAYCnb5uHd88efOvq5kEQRBgAAi8v6dMVe1nHT59Uraz7xdfPvpyu2oBRhAACwWA1akHWuprj/52q++IyFnxtzUoQBAMBSMGhBdr/H1fyo/dd+/bYV3ffTFVtvijAAAFhqBi3InvHutbcdsVvv/RRk60QRBgAArO32m2r+3r/2Hq8g65siDAAAlrJ1Xcjj2R+o+b/+ueZVt/beV0E2L0UYAABMi0EKsofuU/NWD6j5iN2HM7cpoggDAADm98qv13zLNTV/96O999UVW4siDAAAptEgXbENNqq5cxGPb7+v5ku+tbDzW8IUYQAAMO1mCrJly3uP7VzOfrudaj7hlTVfe3H3fXXFkijCAACAGYMu4nHgCTVvtnXNR+1Vc9N033eKCzJFGAAAsP7+73c6npQaP/XskU9l0inCAACAtQ3aFdtw45pfdVrN11xY848+3X3fKeuKKcIAAID5DVqQbf3Amnd9a81nHFZzs6b7vlNQkCnCAACA4Xn8a2q+671qPua5o5/LhFCEAQAA/Ru0K9bp1d+o+bcX1HzRqQszt0VCEQYAAKybQQuyzjGP6+iQffFPal69cuHmN6EUYQAAwPobtCB7Wse1YnfZsuZP7rX22CVGEQYAACysQQuy1/5XzSt+3ZEvX/i5TQBFGAAAMDz9FGSd25/y5po333a4cxsTRRgAADA5nthxfVjnvceWkI3GPQEAAGBKzHTFkiV7D7B+KMIAAIDRm+KCTBEGAACM15QVZK4JAwAAGCGdMAAAYHJ0dsWWKJ0wAACAEVKEAQAAjJAiDAAAYIQUYQAAACOkCAMAABghRRgAAMAIKcIAAABGSBEGAAAwQoowAACAEVKEAQAAjJAiDAAAYIQUYQAAACOkCAMAABghRRgAAMAIKcIAAABGSBEGAAAwQoowAACAEVKEAQAAjJAiDAAAYIQUYQAAACOkCAMAABghRRgAAMAIKcIAAABGSBEGAAAwQoowAACAEVKEAQAAjJAiDAAAYIQUYQAAACOkCAMAABghRRgAAMAIKcIAAABGSBEGAAAwQoowAACAEdpo3BOYZDfeeOO4pwAAAIzRMGqC0jTNgr/pYldKuW+Sy8c9DwAAYGLcr2ma3yzEGynCuiillCTbJ7lp3HNJskVaBeH9MhnzYXFzPLFQHEssJMcTC8WxxEKZfSxtkWR5s0DFk9MRu2j/j7sgVe76atWDSZKbmqZxfiTrxfHEQnEssZAcTywUxxILpcuxtKDHk4U5AAAARkgRBgAAMEKKsMl3e5K/bf8J68vxxEJxLLGQHE8sFMcSC2Wox5KFOQAAAEZIJwwAAGCEFGEAAAAjpAgDAAAYIUUYAADACCnCxqSUsmsp5cullOWllKaU8rxZr5dSymHt128tpXyzlPKIWWO2LqUcW0pZ0X4cW0rZarTfhHHr41g6ur298/HdWWM2LaV8qJRyTSnl5lLKl0op9xvtN2HcSimHllJ+UEq5qZRydSnlpFLKjrPG9DxWSik7tI/Jm9vjPlhK2WS034Zx6vNY+maX36bPzxrj7zlSSnl9KeWnpZQb24+zSynP6njd7xJ96eNYGtnvkiJsfO6W5CdJ3jjH63+V5M3t1x+X5Mokp5VStugY89kkj07yzPbj0UmOHdaEmVi9jqUk+VqS+3Q89p71+vuT7Jdk/yRPSbJ5kpNLKRsu+GyZZE9L8pEkT0yyZ5KNkpxaSrlbx5h5j5X2n19J67h8SnvcC5L884i+A5Ohn2MpSY7InX+bXjfrdX/PkSSXJ3lbkse2H99I8sWO/zjtd4l+9TqWklH9LjVN4zHmR5ImyfM6npckVyR5a8e2TZPckOR17ecPa+/3hI4xT2xv23Hc38ljMo6l9rajk5w0zz5bJlmZ5MUd27ZPsjrJXuP+Th7jeyS5d/uY2rX9vOexkuRZ7efbd4zZP8ltSe4+7u/kMRnHUnvbN5O8f559/D3nMecjyXVJXuV3yWN9HzPHUjuP7HdJJ2wyPSjJdklOndnQNM3tSf4zyR+2Nz0pyYqmab7XMea7SVZ0jIEZu7VPCfplKeWIUso2Ha/tkmTj3Pl4W57kvDiWpt2W7T+va//Zz7HypCTntbfP+Hpa/yFpl6HOlkk2+1ia8dL2qWE/L6X806yzPfw9x1pKKRuWUvZPq6t1dvwusY66HEszRvK7tNF6zJ3h2a7951Wztl+V5AEdY67usu/VHftDknw1yb8nuSytAv9dSb5RStmlXdxvl2Rl0zTXz9rvqjiWplYppSR5X5JvN01zXntzP8fKdpn129U0zfWllJVxPE2lOY6lJPlMkkvSOt3+kUnek2SntE5fTPw9R4dSyh+k9Q/luyT5XZL9mqY5v5Ty6PhdYgBzHUvtl0f2u6QIm2zNrOdl1rbZr3cbw5Rrmua4jqfnlVJ+mFZBtk+S/5hnV8fSdPtwkkeldf1EL36bmE/XY6lpmiM6np5XSrkoyQ9LKY9pmubcmWFd3s+xNJ0uTOvam63Sup7rU6WUp80z3u8Sc+l6LDVNc/4of5ecjjiZrmz/Obui3ib1v+RcmWTbLvveO2t30OB/NU1zRVpF2EPam65MskkpZetZQzuPN6ZIKeVDSfZNsnvTNJd3vNTPsXJlZv12tcdvHMfT1JnnWOrm3CSrcuffJn/PkSRpmmZl0zS/aprmh03THJrWglR/Hr9LDGieY6mbof0uKcIm00wbdKb1mfYyqk9L8p32prOTbFlKeXzHmCekdd79dwJzKKXcM8n901r8JUnOSesHpvN4u09abXjH0hQpLR9O8vwkT2+a5pJZQ/o5Vs5O8sj29hnPSHJ7e3+mQB/HUjePSOsfxTO/Tf6eYz4lrWu6/C6xvmaOpW6G9rtU2qt6MGKllM2T/F776Y/SWo7+zCTXNU3zP6WUtyY5NMnBSS5KsizJbmmtvHJT+z2+mtYKQDNLZ348yWVN0zxnVN+D8ZvvWGo/DktyQlo/IA9McniSHZI8rONY+tckz05yUHuff0pyzyS7NE2zejTfhHErpXw0yUuSPDet0zVmrGia5tb2mHmPlfZS0D9O678I/mWSe6Su0Pmno/kmjFuvY6mU8uAkL01ySpJrkjw8reXCb03yuJnfHX/PkSSllMPTur7510m2SGtlw7cleWbTNKf5XaJf8x1LSf47o/xdGveykNP6SKugaro8jm6/XtL6x/MVaS2h+p9JHjnrPe6R5NNJbmw/Pp1kq3F/N4/JOZaSbJbWClBXp7WE72Xt7fef9R53SfKhJNcmuSXJl2eP8Vj6jzmOoybJQYMcK2kV+Se3X7+2PX7TcX8/j8k5ltLqxv9n+/i4PcmvknwgyT1mvY+/5zyS5BNJLm0fK1cnOT3Jnh2v+13yWO9jadS/SzphAAAAI+SaMAAAgBFShAEAAIyQIgwAAGCEFGEAAAAjpAgDAAAYIUUYAADACCnCAAAARkgRBgAAMEKKMACmWinl0lLKm8Y9DwCmhyIMgKlQSjmolHJDl5cel+TjI/h8xR4ASZKNxj0BABinpml+O+45DKKUsknTNCvHPQ8A1p1OGAAjVUr5Zinlg6WU/1dKua6UcmUp5bA+992ylPLxUsrVpZQbSynfKKXs1PH6TqWUM0spN7VfP6eU8thSym5JjkqyZSmlaT8Oa+9zpw5V+7XXlVJOLqXcUkq5oJTypFLK77XnfnMp5exSyoM79nlwKeWLpZSrSim/K6X8oJTyR53fOckDkvzLzOd3vPaCUsrPSym3t+fyF7O+86WllLeXUo4upaxIckQpZZNSyodLKVeUUm5rjzl0oP8jABgbRRgA4/CKJDcneUKSv0ryjlLKnvPtUEopSb6SZLskeyfZJcm5Sc4opdyjPewzSS5P6xTDXZL8Q5JVSb6T5E1Jbkxyn/bjn+b5uL9JckySRyf5RZLPJvlYkvckeWx7zIc7xm+e5JQkf5Rk5yRfT/LlUsoO7def357XOzo+P6WUXZIcn+TzSf4gyWFJ3lVKOWjWfP4yyXnt7/SuJH+WZN8kL0qyY5IDk1w6z/cBYII4HRGAcfhp0zR/284XlVLemGSPJKfNs8/uaRUq2zRNc3t721tKKc9L8sK0ruvaIcl7m6b5xcx7z+zc7iI1TdNc2cf8jmqa5vj2fv+Y5Owk72qa5uvtbR9Iq7OWtN70J0l+0rH/20sp+6VVKH24aZrrSimrk9w06/PfnOSMpmne1X7+y1LKw9Mquo7uGPeNpmn+t2hsF3cXJfl20zRNksv6+E4ATAidMADG4aeznl+RZJse++ySVsfp2vYpf78rpfwuyYOSzJwa+L4kR5ZSTi+lvK3zlMH1mN9V7T9/NmvbXUopd0+SUsrd2qdXnl9KuaE9r4emVRTO52FJzpq17awkDymlbNix7YezxhydVpfuwvapnc/o+Y0AmBiKMADGYdWs5016/520QVrF2qNnPXZM8t4kaZrmsCSPSOu0xacnOb/dkVqf+TXzbJuZ83uTvCDJXyd5anteP0uySY/PKR3v1blttps7nzRNc25axeffJNksyfGllC/0+CwAJoTTEQFYLM5N63qwO5qmuXSuQU3T/DLJL9NaBONzSQ5OcmKSlUk2nGu/9fTUJEc3TXNikpRSNk/ywFljun3++UmeMmvbHyb5ZdM0q+f7wKZpbkxyXJLj2gXY10op92ia5rp1+woAjIpOGACLxelpXZt1Uillr1LKA0spf1hKeXd7BcTN2isG7lZKeUAp5clpLdBxQXv/S5NsXkrZo5Ryr1LKXRdwbr9K8vxSyqPbqzV+Nmv/HXtpkl1LKfctpdyrve2fk+xRSvmbUsrvl1JekeSNmX/RkJRSDiml7F9KeWgp5feT/HGSK5N0uw8aABNGEQbAotBegGLvJN9K8sm0ul2fT6vjdFWS1Unumdaqhr9Ma9XBryZ5Z3v/7yT5t7S6R79Na1XGhXJIkuvTWoXxy2mtjnjurDHvaM/14vbnz5xW+KIk+6e1+uHfJXlH0zRH9/i83yV5a1rXiv2g/b57N02zZr2/CQBDV1p/pwEAADAKOmEAAAAjpAgDYCKUUl7aufT8rMfPxz0/AFgoTkcEYCKUUrZIsu0cL69qmsYNiQFYEhRhAAAAI+R0RAAAgBFShAEAAIyQIgwAAGCEFGEAAAAjpAgDAAAYIUUYAADACCnCAAAARuj/AynWzvtaINZDAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x165929970b8>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "cvresult = pd.DataFrame.from_csv('my_preds4_2_3_699.csv')\n",
    "\n",
    "cvresult = cvresult.iloc[100:]\n",
    "# plot\n",
    "test_means = cvresult['test-mlogloss-mean']\n",
    "test_stds = cvresult['test-mlogloss-std'] \n",
    "        \n",
    "train_means = cvresult['train-mlogloss-mean']\n",
    "train_stds = cvresult['train-mlogloss-std'] \n",
    "\n",
    "x_axis = range(100,cvresult.shape[0]+100)\n",
    "        \n",
    "fig = pyplot.figure(figsize=(10, 10), dpi=100)\n",
    "pyplot.errorbar(x_axis, test_means, yerr=test_stds ,label='Test')\n",
    "pyplot.errorbar(x_axis, train_means, yerr=train_stds ,label='Train')\n",
    "pyplot.title(\"XGBoost n_estimators vs Log Loss\")\n",
    "pyplot.xlabel( 'n_estimators' )\n",
    "pyplot.ylabel( 'Log Loss' )\n",
    "pyplot.savefig( 'n_estimators_detail4_2_3_699.png' )\n",
    "\n",
    "pyplot.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "通过测试发现，还是340左右"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 行列重采样参数调整"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 77,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'colsample_bytree': [0.6, 0.7, 0.8, 0.9],\n",
       " 'subsample': [0.3, 0.4, 0.5, 0.6, 0.7, 0.8]}"
      ]
     },
     "execution_count": 77,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#max_depth 建议3-10， min_child_weight=1／sqrt(ratio_rare_event) =5.5\n",
    "subsample = [i/10.0 for i in range(3,9)]\n",
    "colsample_bytree = [i/10.0 for i in range(6,10)]\n",
    "param_test3_1 = dict(subsample=subsample, colsample_bytree=colsample_bytree)\n",
    "param_test3_1\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 85,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "D:\\anaconda\\lib\\site-packages\\sklearn\\model_selection\\_search.py:761: DeprecationWarning: The grid_scores_ attribute was deprecated in version 0.18 in favor of the more elaborate cv_results_ attribute. The grid_scores_ attribute will not be available from 0.20\n",
      "  DeprecationWarning)\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "([mean: -0.58984, std: 0.00362, params: {'colsample_bytree': 0.6, 'subsample': 0.3},\n",
       "  mean: -0.58772, std: 0.00396, params: {'colsample_bytree': 0.6, 'subsample': 0.4},\n",
       "  mean: -0.58692, std: 0.00369, params: {'colsample_bytree': 0.6, 'subsample': 0.5},\n",
       "  mean: -0.58671, std: 0.00380, params: {'colsample_bytree': 0.6, 'subsample': 0.6},\n",
       "  mean: -0.58693, std: 0.00317, params: {'colsample_bytree': 0.6, 'subsample': 0.7},\n",
       "  mean: -0.58600, std: 0.00386, params: {'colsample_bytree': 0.6, 'subsample': 0.8},\n",
       "  mean: -0.58975, std: 0.00284, params: {'colsample_bytree': 0.7, 'subsample': 0.3},\n",
       "  mean: -0.58796, std: 0.00357, params: {'colsample_bytree': 0.7, 'subsample': 0.4},\n",
       "  mean: -0.58674, std: 0.00293, params: {'colsample_bytree': 0.7, 'subsample': 0.5},\n",
       "  mean: -0.58636, std: 0.00376, params: {'colsample_bytree': 0.7, 'subsample': 0.6},\n",
       "  mean: -0.58558, std: 0.00400, params: {'colsample_bytree': 0.7, 'subsample': 0.7},\n",
       "  mean: -0.58574, std: 0.00373, params: {'colsample_bytree': 0.7, 'subsample': 0.8},\n",
       "  mean: -0.58922, std: 0.00320, params: {'colsample_bytree': 0.8, 'subsample': 0.3},\n",
       "  mean: -0.58808, std: 0.00392, params: {'colsample_bytree': 0.8, 'subsample': 0.4},\n",
       "  mean: -0.58720, std: 0.00300, params: {'colsample_bytree': 0.8, 'subsample': 0.5},\n",
       "  mean: -0.58586, std: 0.00331, params: {'colsample_bytree': 0.8, 'subsample': 0.6},\n",
       "  mean: -0.58523, std: 0.00338, params: {'colsample_bytree': 0.8, 'subsample': 0.7},\n",
       "  mean: -0.58557, std: 0.00340, params: {'colsample_bytree': 0.8, 'subsample': 0.8},\n",
       "  mean: -0.58903, std: 0.00331, params: {'colsample_bytree': 0.9, 'subsample': 0.3},\n",
       "  mean: -0.58740, std: 0.00372, params: {'colsample_bytree': 0.9, 'subsample': 0.4},\n",
       "  mean: -0.58642, std: 0.00361, params: {'colsample_bytree': 0.9, 'subsample': 0.5},\n",
       "  mean: -0.58584, std: 0.00367, params: {'colsample_bytree': 0.9, 'subsample': 0.6},\n",
       "  mean: -0.58535, std: 0.00370, params: {'colsample_bytree': 0.9, 'subsample': 0.7},\n",
       "  mean: -0.58543, std: 0.00332, params: {'colsample_bytree': 0.9, 'subsample': 0.8}],\n",
       " {'colsample_bytree': 0.8, 'subsample': 0.7},\n",
       " -0.5852302830815046)"
      ]
     },
     "execution_count": 85,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "xgb3_1 = XGBClassifier(\n",
    "        learning_rate =0.1,\n",
    "        n_estimators=340,  #第二轮参数调整得到的n_estimators最优值\n",
    "        max_depth=4,\n",
    "        min_child_weight=1,\n",
    "        gamma=0,\n",
    "        subsample=0.7,\n",
    "        colsample_bytree=0.8,\n",
    "        colsample_bylevel = 0.7,\n",
    "        objective= 'multi:softprob',\n",
    "        seed=3)\n",
    "\n",
    "\n",
    "gsearch3_1 = GridSearchCV(xgb3_1, param_grid = param_test3_1, scoring='neg_log_loss',n_jobs=-1, cv=kfold)\n",
    "gsearch3_1.fit(X_train , y_train)\n",
    "\n",
    "gsearch3_1.grid_scores_, gsearch3_1.best_params_,     gsearch3_1.best_score_\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 86,
   "metadata": {
    "scrolled": false
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "D:\\anaconda\\lib\\site-packages\\sklearn\\utils\\deprecation.py:122: FutureWarning: You are accessing a training score ('mean_train_score'), which will not be available by default any more in 0.21. If you need training scores, please set return_train_score=True\n",
      "  warnings.warn(*warn_args, **warn_kwargs)\n",
      "D:\\anaconda\\lib\\site-packages\\sklearn\\utils\\deprecation.py:122: FutureWarning: You are accessing a training score ('split0_train_score'), which will not be available by default any more in 0.21. If you need training scores, please set return_train_score=True\n",
      "  warnings.warn(*warn_args, **warn_kwargs)\n",
      "D:\\anaconda\\lib\\site-packages\\sklearn\\utils\\deprecation.py:122: FutureWarning: You are accessing a training score ('split1_train_score'), which will not be available by default any more in 0.21. If you need training scores, please set return_train_score=True\n",
      "  warnings.warn(*warn_args, **warn_kwargs)\n",
      "D:\\anaconda\\lib\\site-packages\\sklearn\\utils\\deprecation.py:122: FutureWarning: You are accessing a training score ('split2_train_score'), which will not be available by default any more in 0.21. If you need training scores, please set return_train_score=True\n",
      "  warnings.warn(*warn_args, **warn_kwargs)\n",
      "D:\\anaconda\\lib\\site-packages\\sklearn\\utils\\deprecation.py:122: FutureWarning: You are accessing a training score ('split3_train_score'), which will not be available by default any more in 0.21. If you need training scores, please set return_train_score=True\n",
      "  warnings.warn(*warn_args, **warn_kwargs)\n",
      "D:\\anaconda\\lib\\site-packages\\sklearn\\utils\\deprecation.py:122: FutureWarning: You are accessing a training score ('split4_train_score'), which will not be available by default any more in 0.21. If you need training scores, please set return_train_score=True\n",
      "  warnings.warn(*warn_args, **warn_kwargs)\n",
      "D:\\anaconda\\lib\\site-packages\\sklearn\\utils\\deprecation.py:122: FutureWarning: You are accessing a training score ('std_train_score'), which will not be available by default any more in 0.21. If you need training scores, please set return_train_score=True\n",
      "  warnings.warn(*warn_args, **warn_kwargs)\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "{'mean_fit_time': array([ 265.18794827,  293.7306231 ,  313.64115152,  317.87461348,\n",
       "         316.30303497,  314.71881628,  291.68117528,  324.81926641,\n",
       "         348.58304567,  353.94028392,  355.25717978,  350.1594367 ,\n",
       "         334.60288196,  375.9426435 ,  387.45164027,  390.71315455,\n",
       "         388.80516891,  382.96804833,  342.16938844,  383.03823709,\n",
       "         418.23167453,  448.35708041,  429.50592628,  398.0375113 ]),\n",
       " 'mean_score_time': array([ 0.72332335,  0.7126956 ,  0.71449947,  0.7114922 ,  0.706078  ,\n",
       "         0.70407224,  0.70687938,  0.72472715,  0.7018661 ,  0.72552938,\n",
       "         0.71289601,  0.78909659,  0.76684055,  0.72733436,  0.70928593,\n",
       "         0.8003264 ,  0.70567603,  0.7072803 ,  0.71289539,  0.71470046,\n",
       "         0.70266824,  0.71068964,  0.72833738,  0.5811451 ]),\n",
       " 'mean_test_score': array([-0.58984465, -0.58772316, -0.58691763, -0.58670963, -0.5869294 ,\n",
       "        -0.58599852, -0.58975147, -0.58796206, -0.58674477, -0.58636311,\n",
       "        -0.58557669, -0.58573579, -0.58922177, -0.58807827, -0.58720228,\n",
       "        -0.5858615 , -0.58523028, -0.5855747 , -0.58902704, -0.58740407,\n",
       "        -0.58642066, -0.58584256, -0.58535164, -0.58542838]),\n",
       " 'mean_train_score': array([-0.53034104, -0.52780182, -0.5273331 , -0.52637034, -0.52702307,\n",
       "        -0.52760019, -0.52813552, -0.52636012, -0.52497527, -0.52453624,\n",
       "        -0.52449971, -0.52555442, -0.52566003, -0.52406664, -0.52291371,\n",
       "        -0.52270534, -0.52258242, -0.52407067, -0.52356249, -0.52187786,\n",
       "        -0.52144739, -0.52098592, -0.52148916, -0.52242707]),\n",
       " 'param_colsample_bytree': masked_array(data = [0.6 0.6 0.6 0.6 0.6 0.6 0.7 0.7 0.7 0.7 0.7 0.7 0.8 0.8 0.8 0.8 0.8 0.8\n",
       "  0.9 0.9 0.9 0.9 0.9 0.9],\n",
       "              mask = [False False False False False False False False False False False False\n",
       "  False False False False False False False False False False False False],\n",
       "        fill_value = ?),\n",
       " 'param_subsample': masked_array(data = [0.3 0.4 0.5 0.6 0.7 0.8 0.3 0.4 0.5 0.6 0.7 0.8 0.3 0.4 0.5 0.6 0.7 0.8\n",
       "  0.3 0.4 0.5 0.6 0.7 0.8],\n",
       "              mask = [False False False False False False False False False False False False\n",
       "  False False False False False False False False False False False False],\n",
       "        fill_value = ?),\n",
       " 'params': [{'colsample_bytree': 0.6, 'subsample': 0.3},\n",
       "  {'colsample_bytree': 0.6, 'subsample': 0.4},\n",
       "  {'colsample_bytree': 0.6, 'subsample': 0.5},\n",
       "  {'colsample_bytree': 0.6, 'subsample': 0.6},\n",
       "  {'colsample_bytree': 0.6, 'subsample': 0.7},\n",
       "  {'colsample_bytree': 0.6, 'subsample': 0.8},\n",
       "  {'colsample_bytree': 0.7, 'subsample': 0.3},\n",
       "  {'colsample_bytree': 0.7, 'subsample': 0.4},\n",
       "  {'colsample_bytree': 0.7, 'subsample': 0.5},\n",
       "  {'colsample_bytree': 0.7, 'subsample': 0.6},\n",
       "  {'colsample_bytree': 0.7, 'subsample': 0.7},\n",
       "  {'colsample_bytree': 0.7, 'subsample': 0.8},\n",
       "  {'colsample_bytree': 0.8, 'subsample': 0.3},\n",
       "  {'colsample_bytree': 0.8, 'subsample': 0.4},\n",
       "  {'colsample_bytree': 0.8, 'subsample': 0.5},\n",
       "  {'colsample_bytree': 0.8, 'subsample': 0.6},\n",
       "  {'colsample_bytree': 0.8, 'subsample': 0.7},\n",
       "  {'colsample_bytree': 0.8, 'subsample': 0.8},\n",
       "  {'colsample_bytree': 0.9, 'subsample': 0.3},\n",
       "  {'colsample_bytree': 0.9, 'subsample': 0.4},\n",
       "  {'colsample_bytree': 0.9, 'subsample': 0.5},\n",
       "  {'colsample_bytree': 0.9, 'subsample': 0.6},\n",
       "  {'colsample_bytree': 0.9, 'subsample': 0.7},\n",
       "  {'colsample_bytree': 0.9, 'subsample': 0.8}],\n",
       " 'rank_test_score': array([24, 18, 14, 12, 15,  9, 23, 19, 13, 10,  5,  6, 22, 20, 16,  8,  1,\n",
       "         4, 21, 17, 11,  7,  2,  3]),\n",
       " 'split0_test_score': array([-0.58455687, -0.58108539, -0.58103466, -0.58110304, -0.58171937,\n",
       "        -0.58044932, -0.5848076 , -0.5828937 , -0.58157764, -0.58021157,\n",
       "        -0.57917317, -0.5798642 , -0.58375048, -0.5818616 , -0.5822169 ,\n",
       "        -0.58027584, -0.5799731 , -0.5802152 , -0.58302903, -0.58052262,\n",
       "        -0.58033067, -0.57962481, -0.57867884, -0.58019666]),\n",
       " 'split0_train_score': array([-0.53151306, -0.52837964, -0.52870423, -0.52882176, -0.52838765,\n",
       "        -0.52895815, -0.53055548, -0.52818301, -0.5268853 , -0.52576805,\n",
       "        -0.52552574, -0.52651494, -0.52777722, -0.52559064, -0.52485652,\n",
       "        -0.5237521 , -0.52355827, -0.52598857, -0.52548356, -0.5238144 ,\n",
       "        -0.5225538 , -0.52265591, -0.52307304, -0.52331832]),\n",
       " 'split1_test_score': array([-0.58810009, -0.58564934, -0.58549914, -0.58444223, -0.58529023,\n",
       "        -0.58360228, -0.58878466, -0.58526613, -0.5859539 , -0.58453298,\n",
       "        -0.58363694, -0.58376711, -0.58837488, -0.58565881, -0.58649707,\n",
       "        -0.58555872, -0.58359141, -0.58407666, -0.58844214, -0.58680791,\n",
       "        -0.58514662, -0.5846553 , -0.58485275, -0.58411465]),\n",
       " 'split1_train_score': array([-0.53092995, -0.52805249, -0.52752584, -0.52604586, -0.52714782,\n",
       "        -0.5282056 , -0.52756271, -0.5261204 , -0.5247611 , -0.52450183,\n",
       "        -0.52463856, -0.52451585, -0.52496446, -0.5251473 , -0.52353862,\n",
       "        -0.52303319, -0.52211939, -0.52397528, -0.52373231, -0.52231178,\n",
       "        -0.52235547, -0.52092899, -0.52200311, -0.52278749]),\n",
       " 'split2_test_score': array([-0.58911388, -0.58885578, -0.58657934, -0.58633055, -0.58741338,\n",
       "        -0.58529123, -0.59035772, -0.58859247, -0.58713992, -0.58667906,\n",
       "        -0.58552887, -0.58582602, -0.58932304, -0.5887798 , -0.58717035,\n",
       "        -0.58506087, -0.58504328, -0.58512135, -0.58976487, -0.58866695,\n",
       "        -0.58651193, -0.58630792, -0.58565758, -0.58491198]),\n",
       " 'split2_train_score': array([-0.53028301, -0.52833461, -0.52790887, -0.52582373, -0.52678616,\n",
       "        -0.52764153, -0.52727081, -0.52585007, -0.52462983, -0.5240479 ,\n",
       "        -0.52364875, -0.52628363, -0.52584425, -0.52411045, -0.52284356,\n",
       "        -0.52222183, -0.52309347, -0.5237661 , -0.5229434 , -0.52078208,\n",
       "        -0.52087118, -0.52104446, -0.52127582, -0.52224832]),\n",
       " 'split3_test_score': array([-0.59509816, -0.59144286, -0.58971462, -0.58997136, -0.58994666,\n",
       "        -0.5905716 , -0.59195613, -0.58998636, -0.58938409, -0.59028053,\n",
       "        -0.59004078, -0.59040156, -0.59160459, -0.59158813, -0.58877285,\n",
       "        -0.58865009, -0.58965195, -0.58918391, -0.59254905, -0.59103432,\n",
       "        -0.58983784, -0.58811651, -0.58891551, -0.58955926]),\n",
       " 'split3_train_score': array([-0.53072828, -0.5274064 , -0.52621593, -0.52539127, -0.52668668,\n",
       "        -0.52731486, -0.52785737, -0.52558303, -0.52450549, -0.52378662,\n",
       "        -0.52447708, -0.52534048, -0.52457951, -0.52257748, -0.52147146,\n",
       "        -0.52261329, -0.52256747, -0.52396481, -0.52317517, -0.52189849,\n",
       "        -0.52109517, -0.5205385 , -0.52086309, -0.52237087]),\n",
       " 'split4_test_score': array([-0.592355  , -0.59158363, -0.59176184, -0.59170247, -0.5902784 ,\n",
       "        -0.59007942, -0.59285216, -0.59307318, -0.58966921, -0.59011255,\n",
       "        -0.58950488, -0.588821  , -0.59305704, -0.59250436, -0.59135548,\n",
       "        -0.58976315, -0.58789248, -0.58927752, -0.59135083, -0.58998934,\n",
       "        -0.5902774 , -0.59050967, -0.58865454, -0.58836026]),\n",
       " 'split4_train_score': array([-0.52825089, -0.52683594, -0.52631062, -0.52576905, -0.52610701,\n",
       "        -0.52588079, -0.52743122, -0.52606412, -0.52409463, -0.52457681,\n",
       "        -0.52420841, -0.52511722, -0.52513471, -0.52290732, -0.52185841,\n",
       "        -0.52190631, -0.52157351, -0.52265857, -0.522478  , -0.52058255,\n",
       "        -0.52036135, -0.51976172, -0.52023072, -0.52141034]),\n",
       " 'std_fit_time': array([  1.50306479,   2.10343189,   0.95422565,   2.00001972,\n",
       "          0.57742142,   1.17474721,   0.91512385,   3.63749108,\n",
       "          9.27902589,   4.56891275,   3.01746708,   4.26347606,\n",
       "         10.72714637,   3.86578464,   9.09430713,   3.71215258,\n",
       "          0.85608511,   2.8682612 ,   1.04388783,   0.70751018,\n",
       "         13.74485745,   3.47701528,  14.2926559 ,  15.49597033]),\n",
       " 'std_score_time': array([ 0.02128444,  0.01731327,  0.01644601,  0.00813593,  0.00790507,\n",
       "         0.01599772,  0.00560081,  0.03827196,  0.00570764,  0.03788208,\n",
       "         0.0100465 ,  0.12970941,  0.05046084,  0.02326235,  0.00875482,\n",
       "         0.15286845,  0.00669289,  0.01475822,  0.00658974,  0.00710083,\n",
       "         0.0056153 ,  0.0079563 ,  0.03777649,  0.13147797]),\n",
       " 'std_test_score': array([ 0.00361777,  0.00395937,  0.00368843,  0.00380218,  0.00317443,\n",
       "         0.0038627 ,  0.00283614,  0.0035657 ,  0.00293255,  0.00376019,\n",
       "         0.00400322,  0.00373226,  0.00319525,  0.00392176,  0.00300219,\n",
       "         0.00331452,  0.00337635,  0.00340325,  0.00330662,  0.00371938,\n",
       "         0.00361305,  0.00366707,  0.00370081,  0.00331735]),\n",
       " 'std_train_score': array([ 0.00111736,  0.00059495,  0.00095312,  0.00124368,  0.00075975,\n",
       "         0.0010247 ,  0.00122519,  0.00093081,  0.00098078,  0.00068126,\n",
       "         0.00061343,  0.0007443 ,  0.00113523,  0.00118776,  0.00121357,\n",
       "         0.00064558,  0.00069943,  0.00107568,  0.001042  ,  0.00116769,\n",
       "         0.00085841,  0.00094809,  0.00097918,  0.00063136])}"
      ]
     },
     "execution_count": 86,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "gsearch3_1.cv_results_"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 87,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Best: -0.585230 using {'colsample_bytree': 0.8, 'subsample': 0.7}\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "D:\\anaconda\\lib\\site-packages\\sklearn\\utils\\deprecation.py:122: FutureWarning: You are accessing a training score ('mean_train_score'), which will not be available by default any more in 0.21. If you need training scores, please set return_train_score=True\n",
      "  warnings.warn(*warn_args, **warn_kwargs)\n",
      "D:\\anaconda\\lib\\site-packages\\sklearn\\utils\\deprecation.py:122: FutureWarning: You are accessing a training score ('std_train_score'), which will not be available by default any more in 0.21. If you need training scores, please set return_train_score=True\n",
      "  warnings.warn(*warn_args, **warn_kwargs)\n",
      "D:\\anaconda\\lib\\site-packages\\sklearn\\utils\\deprecation.py:122: FutureWarning: You are accessing a training score ('split0_train_score'), which will not be available by default any more in 0.21. If you need training scores, please set return_train_score=True\n",
      "  warnings.warn(*warn_args, **warn_kwargs)\n",
      "D:\\anaconda\\lib\\site-packages\\sklearn\\utils\\deprecation.py:122: FutureWarning: You are accessing a training score ('split1_train_score'), which will not be available by default any more in 0.21. If you need training scores, please set return_train_score=True\n",
      "  warnings.warn(*warn_args, **warn_kwargs)\n",
      "D:\\anaconda\\lib\\site-packages\\sklearn\\utils\\deprecation.py:122: FutureWarning: You are accessing a training score ('split2_train_score'), which will not be available by default any more in 0.21. If you need training scores, please set return_train_score=True\n",
      "  warnings.warn(*warn_args, **warn_kwargs)\n",
      "D:\\anaconda\\lib\\site-packages\\sklearn\\utils\\deprecation.py:122: FutureWarning: You are accessing a training score ('split3_train_score'), which will not be available by default any more in 0.21. If you need training scores, please set return_train_score=True\n",
      "  warnings.warn(*warn_args, **warn_kwargs)\n",
      "D:\\anaconda\\lib\\site-packages\\sklearn\\utils\\deprecation.py:122: FutureWarning: You are accessing a training score ('split4_train_score'), which will not be available by default any more in 0.21. If you need training scores, please set return_train_score=True\n",
      "  warnings.warn(*warn_args, **warn_kwargs)\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAZIAAAEKCAYAAAA4t9PUAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMS4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvNQv5yAAAIABJREFUeJzs3XdcldUfwPHPAS6yBBVQVFTErQhqbnPvn+beaVrOytTKvbUsTTNXOSstB6bm3rkzt+ECF4iCIiIICoKs8/vjXvGSoBe4VxHP+/W6L+B5nnOe81zhfj1bSClRFEVRlMwye90FUBRFUd5sKpAoiqIoWaICiaIoipIlKpAoiqIoWaICiaIoipIlKpAoiqIoWaICiaIoipIlKpAoiqIoWaICiaIoipIlFqbMXAjRApgLmAPLpJTT/3O+DzATuK07tEBKuUx3bgbQSnf8KynlWt3x4oA3kA84C/SSUsa/qBxOTk7Szc3NGI+kKIry1jhz5sx9KaXzy64zWSARQpgDPwJNgWDglBBii5TS9z+XrpVSDv5P2lZAFaASkAs4JITYKaV8CMwAfpBSegshFgF9gYUvKoubmxunT582ynMpiqK8LYQQNw25zpRNW9WB61LKAF2NwRtoa2Da8sAhKWWilDIGOAe0EEIIoBGwXnfdCqCdkcutKIqiZIApA0lhIEjv52Ddsf/qKIQ4L4RYL4Qoojt2DmgphLARQjgBDYEigCMQKaVMfEmeiqIoyitiykAi0jj236WGtwJuUkpP4C+0NQyklHuAHcA/wBrgGJBoYJ7amwsxQAhxWghxOiwsLHNPoCiKoryUKTvbg9HWIp5yBe7oXyClDNf7cSna/o+n56YB0wCEEKuBa8B9II8QwkJXK3kuT730S4AlAFWrVlVr5SuvXUJCAsHBwcTFxb3uoihKKlZWVri6uqLRaDKV3pSB5BRQSjfK6jbQDeihf4EQoqCUMkT3YxvAT3fcHMgjpQwXQngCnsAeKaUUQhwAOqHtc+kNbDbhMyiK0QQHB5M7d27c3NzQdvcpyusnpSQ8PJzg4GCKFy+eqTxMFkiklIlCiMHAbrTDf3+RUl4SQkwFTksptwBDhBBt0DZbRQB9dMk1wBHdH9tDoKdev8gowFsI8TXwL/CzqZ5BUYwpLi5OBREl2xFC4OjoSFa6AEw6j0RKuQNtX4f+sYl6348BxqSRLg7tyK208gxAOyJMUd44Kogo2VFWfy/VzPZ0SCn541QQey7dfd1FURRFydZMWiN5kyUlS347HkhIZBzvFMuLo12u110kRVGUbEnVSNJhYW7G950r8TAugYmbL73u4ihKlkVGRvLTTz9lKu2cOXN4/Pix0cqyfPlyBg8e/PILjezgwYO0bt06w+kyWt7AwEBWr16d4fsYKiIigqZNm1KqVCmaNm3KgwcP0rzu1q1bNGvWjHLlylG+fHkCAwNNUh4VSF6gjEtuhjUpzfYLIWw7n+YoY0V5Y2SnQJLTvSiQJCYmpnk8I6ZPn07jxo25du0ajRs3Zvr06Wle98EHHzBixAj8/Pw4efIk+fPnz/K906Katl7kzHIGueRnt6sDEzdfoqa7I06qiUsxgilbL+F756FR8yxfyJ5J71VI9/zo0aPx9/enUqVKNG3alPz58/PHH3/w5MkT2rdvz5QpU4iJiaFLly4EBweTlJTEhAkTCA0N5c6dOzRs2BAnJycOHDiQZv67du1i7NixJCUl4eTkxL59+4iIiOCjjz4iICAAGxsblixZgqenZ6p069atY8qUKZibm+Pg4MDhw4cJDAykV69exMTEALBgwQJq167NwYMHmTRpEgUKFMDHx4cOHTpQsWJF5s6dS2xsLJs2baJEiRL06dMHKysrLl26RGhoKLNnz36uJhITE8Nnn33GhQsXSExMZPLkybRtm/4qTkFBQbRo0YIbN27Qo0cPJk2axIQJE3BycmLo0KEAjBs3jgIFCrB69Wr8/PyoVKkSvXv3Jm/evGzfvp24uDhiYmLYv38/M2fOfO79B1i5ciXz5s0jPj6eGjVq8NNPP2Fubp6qLJs3b+bgwYMA9O7dmwYNGjBjxoxU1/j6+pKYmEjTpk0BsLOzS/fZskoFkvQkJcDpXzCPuMG8Dtto9tsjJmy6yE/vV1Ejb5Q30vTp07l48SI+Pj7s2bOH9evXc/LkSaSUtGnThsOHDxMWFkahQoXYvn07AFFRUTg4ODB79mwOHDiAk5NTmnmHhYXRv39/Dh8+TPHixYmIiABg0qRJVK5cmU2bNrF//34++OADfHx8UqWdOnUqu3fvpnDhwkRGRgKQP39+9u7di5WVFdeuXaN79+4pC6+eO3cOPz8/8uXLh7u7O/369ePkyZPMnTuX+fPnM2fOHEBbKzh06BD+/v40bNiQ69evp7rvtGnTaNSoEb/88guRkZFUr16dJk2aYGtrm+Yznjx5kosXL2JjY0O1atVo1aoVffv2pUOHDgwdOpTk5GS8vb05efIknp6ezJo1i23btgHaprFjx45x/vx58uXLx549e7h27dpz77+zszNr167l6NGjaDQaPvnkE1atWsUHH3xAv379GDRoEFWrViU0NJSCBQsCULBgQe7du/dcea9evUqePHno0KEDN27coEmTJkyfPv25oGQMKpCkx1wDXVfCkga4/TWAkQ2X8vVfQWw9H0Ibr0Kvu3TKG+5FNYdXYc+ePezZs4fKlSsDEB0dzbVr16hbty7Dhw9n1KhRtG7dmrp16xqU3/Hjx6lXr17KhLZ8+fIB8Pfff7NhwwYAGjVqRHh4OFFRUanS1qlThz59+tClSxc6dOgAaFcBGDx4MD4+Ppibm3P16tWU66tVq5byIVqiRAmaNWsGQMWKFVPVlrp06YKZmRmlSpXC3d2dy5cvP/cebNmyhVmzZgHaeT63bt2iXLlyaT5j06ZNcXR0BKBDhw78/fffDBs2DEdHR/79919CQ0OpXLlyyjVppX/6vqT3/p8/f54zZ85QrVo1AGJjY1Oao5YtW5ZmvulJTEzkyJEj/PvvvxQtWpSuXbuyfPly+vbtm6F8DKECyYvkKQqdfoXf2/GR0yy2ug5g0uaL1HJ3xDm3auJS3lxSSsaMGcPAgQOfO3fmzBl27NjBmDFjaNasGRMnTkwjh+fzS6umLuXzqxP997pFixZx4sQJtm/fTqVKlfDx8WH+/PkUKFCAc+fOkZycjJWVVcr1uXI9+9szMzNL+dnMzCxV/8N/7/Pfn6WUbNiwgTJlyrz0+V6UX79+/Vi+fDl3797lo48+Sje9fk0nvfd//vz59O7dm2+//faFZSlQoAAhISEULFiQkJCQNPs+XF1dqVy5Mu7u7gC0a9eO48ePmySQqM72l3GvD02nYua3mZ9L/k1MfBLjN11I8w9EUbKz3Llz8+jRIwCaN2/OL7/8QnR0NAC3b9/m3r173LlzBxsbG3r27Mnw4cM5e/bsc2nTUqtWLQ4dOsSNGzcAUpq26tWrx6pVqwDtiCknJyfs7e1TpfX396dGjRpMnToVJycngoKCiIqKomDBgpiZmfH777+TlJSU4eddt24dycnJ+Pv7ExAQ8FzAaN68OfPnz0/5W/73339fmN/evXuJiIhI6YupU6cOAO3bt2fXrl2cOnWK5s2bAy9/v9J7/xs3bsz69etTmqoiIiK4efP5LUHatGnDihUrAFixYkWafTvVqlXjwYMHKTPW9+/fT/nyac7zzjJVIzFErcFw51+cTszgh3fm8+mJZLacu0PbSmoFe+XN4ejoSJ06dfDw8KBly5b06NGDWrVqAdqO2JUrV3L9+nVGjBiBmZkZGo2GhQu1e8YNGDCAli1bUrBgwTQ7252dnVmyZAkdOnQgOTk5pY9j8uTJfPjhh3h6emJjY5Py4advxIgRXLt2DSkljRs3xsvLi08++YSOHTuybt06GjZsmG6/xYuUKVOG+vXrExoayqJFi1LVagAmTJjAsGHD8PT0REqJm5tbSp9GWt5991169erF9evX6dGjB1WrVgXA0tKShg0bkidPnpT+B09PTywsLPDy8qJPnz7kzZs3VV7NmjXDz8/vufe/fPnyfP311zRr1ozk5GQ0Gg0//vgjxYoVS9VHMnr0aLp06cLPP/9M0aJFWbduHQCnT59m0aJFLFu2DHNzc2bNmkXjxo2RUvLOO+/Qv3//DL+PhhBvw/+sq1atKrO8Q2J8DCxrinx4m0G2szkekZu9n9cjv73Vy9MqCuDn55du+7tiXH369KF169Z06tTJ5PdKTk6mSpUqrFu3jlKlSpn8fqaS1u+nEOKMlLLqy9Kqpi1DWdpCt5UIJPOYBQmPGbvxomriUpS3mK+vLyVLlqRx48ZvdBDJKtW0lRH53KHjz+Ra1ZkNhb1p4teTTT63aV/Z9XWXTFFemRo1avDkyZNUx37//XcqVqz4mkr0vOXLl2cq3e7duxk1alSqY8WLF2fjxo1pXl++fHkCAgIyda+cRAWSjCrVFBqNo+T+r5noVIRJmzXULuFEAdXEpbwlTpw48bqLYDLNmzdP6TBXDKeatjLj3S+hbGs+jPkZr6QLjP1TjeJSFOXtpQJJZpiZQbuFCMcSLLFagO9lPzacvf26S6UoivJaqECSWVb20G01ViKR3+3m8e3Wf7kbpfbiVhTl7aMCSVY4lUJ0WEzJxGuMSV7G6A3nVBOXoihvHRVIsqpsK6g3gk5mByns7826M8Gvu0SKkqbstIy82o8kawzZj+TAgQNUqlQp5WVlZcWmTZtMUh4VSIyhwRhkyWZM0fzGlq0buRMZ+7pLpCjPyU6BJKfLDvuRNGzYEB8fH3x8fNi/fz82NjYpC1wamxr+awxm5oiOS5CLGvBD5PdMXVeKef1aqOXmlfTtHA13Lxg3T5eK0DLtDY5A7Ufytu1Hom/9+vW0bNkSGxubdK/JChVIjMU6L5oea8izpCG9gyay7kRputQs8bpLpSgp1H4kb9d+JPq8vb354osvXnhNVqhAYkwFymPebiFVN3zItZ2juV32NwrnsX7dpVKyoxfUHF4FtR9Jzt+P5KmQkBAuXLhg0omWKpAYmVnFDjy8cYruZ39iyYqZ9B8yQTVxKdmO2o8k5+9H8tQff/xB+/bt0Wg0L8wzK1RnuwnYt/qKEMca9I6Yx+69u153cRQFUPuRvG37kTy1Zs0aunfv/sJnyypVIzEFcwsKfLia8B9q43X0U+5UOEihwkVfd6mUt5zaj+Tt2o8EtP1EQUFB1K9fP8PvX0ao/UheICE5AY1Z5quDoVeO47C6NQFW5Sg3cj/C3HRVSyX7U/uRvDpqP5KMU/uRmECyTObzA5/z7YlvSUhOyFQeBcrU5KzXZMo/Oc/l3z43cgkVRXnd1H4kWqppKx3JMpli9sX4zfc3rj64yvcNviefVb4M51Or/afsDjhJ85u/E36sJo61epqgtIry6qj9SJ5R+5Foqaatl9jqv5Upx6aQzyofcxrOobxj+QzncTv8ISHzmlHRLABN/72YFfLKVFmUN5tq2lKyM9W0ZULvlXiPFS1XIJF8sPMDtgWk3xmXnsKO9gQ1+YmIZFtifu8OjyNMUFJFUZTXQwUSA1RwrIB3K288nDwYc2QMs07NIjE5Y+vltHu3MotdJpPrcSix3n0gOePDGRVFUbIjFUgM5GjtyNJmS+letjsrfFfw8V8fExkXaXB6IQQDunfhGz7C+tYh5L6vTVhaRVGUV0cFkgzQmGkYW2MsU2tP5UzoGbpt78aViCsGpy+Ux5ryrYewOrER4uhs8N1swtIqSmrZafVftYx81hiyjDzAyJEjqVChAuXKlWPIkCEm2y/JpIFECNFCCHFFCHFdCDE6jfN9hBBhQggf3auf3rnvhBCXhBB+Qoh5QrcegRCiqxDivO7cd6Ysf3ral2rP8hbLSUhKoNfOXuwJ3GNw2s5VXdnv/iU+shTJGz+Ge5dfnkhRjCA7BZKcLjssI//PP/9w9OhRzp8/z8WLFzl16hSHDh3K8r3TYrJAIoQwB34EWgLlge5CiLSGPK2VUlbSvZbp0tYG6gCegAdQDagvhHAEZgKNpZQVgAJCiMameobITZuI+eefNM95Onvi3dqb0nlL8+WhL5l3dh5JBvR7CCH4umNVvhRf8jDJEundA+KiXppOUbJKfxn5ESNGMHPmTKpVq4anpyeTJk0CtEurt2rVCi8vLzw8PFi7di3z5s1LWUa+YcOG6ea/a9cuqlSpgpeXF40ba/8sIyIiaNeuHZ6entSsWZPz588/l27dunV4eHjg5eVFvXr1AO0Hcd26dalSpQpVqlThH93f4cGDB6lfvz5dunShdOnSjB49mlWrVlG9enUqVqyIv78/oJ2QOGjQIOrWrUvp0qXTnLEeExPDRx99RLVq1ahcuTKbN7+4heDpMvJlypRJWfJ9woQJzJ07N+WacePGMW/ePEaPHs2RI0eoVKkSP/zwA8uXL6dz58689957KYtMpvX+g3YZ+erVq1OpUiUGDhyY5vIwmzdvpnfv3oB2Gfm0NqwSQhAXF0d8fDxPnjwhISGBAgUKvPAZM8uU80iqA9ellAEAQghvoC3ga0BaCVgBloAANEAo4A5clVKG6a77C+gI7DNu0UHGxxPxy68k3LlDsVWrsCpT+rlrnG2c+aX5L3xz4huWXljK5YjLTK83HXtL+zRyfMbFwYqP36tL//WfsVZ+g/hzAHRbA2aqpfFtMePkDC5HGLc2WjZfWUZVH5XuebWM/Nu1jHytWrVo2LAhBQsWRErJ4MGDTTb83JSfXIWBIL2fg3XH/qujrqlqvRCiCICU8hhwAAjRvXZLKf2A60BZIYSbEMICaAcUSevmQogBQojTQojTYWFhaV3yQsLSkiKLF2FmY0PQwIEk3L2b5nWW5pZMrj2ZCTUncOzOMXps70FA5MsnKHWsUpjcZeoxLaknXN0Fh19LK53yltJfxrxKlSpcvnyZa9euUbFiRf766y9GjRrFkSNHcHBwMCi/Fy0j36tXL+Dly8gvXbo05X/fCQkJ9O/fn4oVK9K5c2d8fZ/9//PpMvK5cuV6bhn5wMDAlOsMWUZ++vTpVKpUiQYNGqQsI5+ep8vIW1tbpywj7+bmlrKM/NP3M6PLyOu///v27UtZRr5SpUrs27cvZcLjsmXLUtb3MsT169fx8/MjODiY27dvs3//fg4fPmxw+owwZY0krbXT/9vTsxVYI6V8IoQYBKwAGgkhSgLlAFfddXuFEPWklIeFEB8Da4Fk4B+0tZTnbyTlEmAJaCckZuYBNAULUmTJYm6+35OggYMotmol5nZ2aV7bpUwXSuYpyRcHv6DHjh588+43NCraKN28hRB826EiTWe35F2rIBoe/BYKVoIyLTJTVOUN86Kaw6uglpHP+cvIb9y4kZo1a2Kn+8xq2bJlSsA3NlPWSIJJXVtwBe7oXyClDJdSPl1rYSnwju779sBxKWW0lDIa2AnU1KXZKqWsIaWsBVwBrpnwGbAqW5bCc+fy5Pp1bg8ZikxIf92tKgWq4N3aGzd7N4YeGMpPPj+RLJPTvb6AvRWT23gwKLIX93OXgz/7w/3r6V6vKFmhlpF/u5aRL1q0KIcOHSIxMZGEhAQOHTr0RjZtnQJKCSGKCyEsgW7AFv0LhBAF9X5sA/jpvr+FtnPdQgihAeo/PSeEyK/7mhf4BMjctmEZYPduHQpOnUrMP/8QMmnyC4fQudi6sKLlCtqUaMPCcwsZemAo0fHR6V7fvnJh6pYrQucHn5AkzGHt+/Ak/V9ARcks/WXk9+7dm7KMfMWKFenUqROPHj3iwoULKR2906ZNY/z48cCzZeTT62zXX0bey8uLrl27AjB58mROnz6Np6cno0ePTncZ+YoVK+Lh4UG9evVSlpFfsWIFNWvW5OrVq1laRr5ly5bpLiOfkJCAp6cnHh4eTJgw4YX5PV1GvlKlSnTs2PG5ZeS7dOmS5jLyP/zww3N5NWvWLM33X38ZeU9PT5o2bUpISAigrfk87ScaPXo0e/fupVSpUuzdu5fRo7WDYk+fPk2/ftrBr506daJEiRJUrFgRLy8vvLy8eO+99zL8PhpESmmyF/A/4CrgD4zTHZsKtNF9/y1wCTiHtk+krO64ObAYbfDwBWbr5blGd8wX6GZIOd555x1pDPfmzZe+ZcrKewsWvPTa5ORkudJ3pfRa4SXbbGwjb0TeSPfa0KhY6Tl5t5zwwwKZPDmPlGt7SZmcbJQyK9mHr6/v6y7CW6N3795y3bp1r+ReSUlJ0svLS169evWV3M9U0vr9BE5LAz5jTTpMSEq5Q0pZWkpZQko5TXdsopRyi+77MVLKClJKLyllQynlZd3xJCnlQCllOSlleSnlF3p5dtcdKy+l9DZl+f/LafCnOLRrx/35C4jc+PxwO31CCN4v9z5Lmy3lQdwDemzvweHgtDu68ttbMaVNBX6768Zx9yHaiYpH55jiERRFMSK1jLyWWv03g2R8PEGDBhFz8hRFlyzGtnbtl6a5E32HYQeGcTniMkOqDKGvR980O/4G/n6Gg1fvcabsGnL7b4P310NJk02TUV6xnLL675uwjHxmZXQZ+ZwkK6v/qkCSCUmPHnHz/Z4k3L5NsdWrsDJg1EdsYiyT/pnEzhs7aVqsKV/X+RobjU2qa+49iqPZD4cpnc+ctWbjEQ/vwMBDkNfNaGVXXp+cEkiUnEktI/+KmefOTZElizGzsyNoQPpzTPRZW1gzo+4Mhlcdzr5b++i5sydBj4JSXZM/txVT23pwMjiO1cW/BSR494R4tTSFoijZlwokmaRxcaHI4kUkR0cTNGAgSS8Y6veUEILeFXqzsPFCQmNC6batG//cSb0Ey3ueBWlRwYUpf8dyu9F8CL0IW4fAW1BzVBTlzaQCSRZYlS1L4XlzeRIQwO2hL55joq924dp4t/Imv01+Pv7rY1ZcWpEypFgIwdftPbCzsuDjk44kNRgLF9bB8YWmfBRFUZRMU4Eki+zqPJ1jcoyQiZMMXqa5iH0RVv1vFY2LNmbW6VmMPjKa2MRYAJzscjG1bQXOB0exKLkdlG0Ne8bDjSOmfBRFUZRMUYHECPJ0aI/T4MFEbdzI/R8NX6bbRmPD9/W/Z0jlIey8sZPeO3tzJ1o7+b+1ZyFaVSzInH3XuVZ7JjiWgHV9ICrYRE+h5HTZaRl5tR9J1hi6H8moUaPw8PBIWcnZVFQgMRKnTz/BoX177i9YQOSfhg8VFELQ37M/CxovIPhRMN22dePU3VMATG1bAXsrDV9sDiCh8++Q+ATW9oSEOFM9hpKDZadAktNlh/1Itm/fztmzZ/Hx8eHEiRPMnDmThw8fZvneaVGBxEiEEBScOgXb2rUJmTiR6L+PZih9Pdd6rG61mrxWeem/pz+r/FaRz9aSr9t5cOF2FIsumkOHxXDnX9jxpep8VzJM7UeSWk7fj8TX15f69etjYWGBra0tXl5e7Nq164XPmGmGTH9/01/GWiLFEImPHkn/Nm3l5SrvyFg/vwynf/TkkRy8b7D0WO4hxx0ZJ+MS4+Snq87IkmO3S987UVLu+0rKSfZSnlxmgtIrpqS/BEXItGkysGcvo75Cpk174f1v3LghK1SoIKWUcvfu3bJ///4yOTlZJiUlyVatWslDhw7J9evXy379+qWkiYyMlFJKWaxYMRkWFpZu3vfu3ZOurq4yICBASilleHi4lFLKwYMHy8mTJ0sppdy3b5/08vKSUkr566+/yk8//VRKKaWHh4cMDg6WUkr54MEDKaWUMTExMjY2Vkop5dWrV+XTv+EDBw5IBwcHeefOHRkXFycLFSokJ06cKKWUcs6cOXLo0KFSSu0SKc2bN5dJSUny6tWrsnDhwjI2NlYeOHBAtmrVSkop5ZgxY+Tvv/+ect9SpUrJ6OjoNJ/v119/lS4uLvL+/fvy8ePHskKFCvLUqVPyxo0bsnLlylJK7VIp7u7u8v79+6nu8zR94cKFU96X9N5/X19f2bp1axkfHy+llPLjjz+WK1askFJK2bdvX3nq1CkppZQODg6pypcnT57nyrx7925Zu3ZtGRMTI8PCwmTx4sXlrFmz0vsnzNISKaZcRv6tZG5nR5HFiwjs2o2ggYNwW+uNxsXF4PR2lnbMbTiXRecWsfDcQvwj/ZnU9DuOB4QzfN05Nn08Cs0dH9g5Cgp4QNEaJnwaJafS3w8DIDo6mmvXrlG3bl2GDx/OqFGjaN26NXXr1jUovxftR7Jhwwbg5fuRdOnShQ4dOgDa/UgGDx6Mj48P5ubmXL16NeX6p/uRAM/tR3LgwIGU6wzZj2TLli3MmjULIGU/kvQmjT7djwRI2Y9k2LBhKfuRhIaGZmo/Enj2/p8/fz5lPxKA2NjYlCXily3L2Pq0zZo149SpU9SuXRtnZ2dq1aqFhYVpPvJVIDEBjYvLs31MBgzU7mOSO7fB6c2EGZ9U+oQy+cow9shYBu3vRd8m45mx6SE/HQpkaMelsKQh/NELBh6G3IYHKiV7cBk79rXeX6r9SHL8fiSgbWobN24cAD169DDZemCqj8RErMqUwVV/jkl8fIbzaFy0MatbrcZWY8uSayOoWvEy8/df49IDM+i2Wrvc/B8fQGLG81bePmo/krdrP5KkpCTCw8MBOH/+POfPn0+pvRmbCiQmZFu7NgW/+irDc0z0lchTgjWt11CzYE2uJC7HtvAmvlx3lnjHstD2Rwg6AbvHmKD0Sk6j9iN5u/YjSUhIoG7dupQvX54BAwawcuVKkzVtvfaO8FfxepWd7Wm5t2CBdh+TefMznUdiUqKcc2aO9FjuIcv91Ep+tfO49sTu8drO97O/G6m0iqmo/UheHbUfScZl2/1IFC2nTz7BoUMH7v/4I5G6jseMMjczZ2iVocyqPwuNTQjewV+wyfcYNJ4ExevDti/g9lkjl1xRlBdR+5FoqWXkXxGZkEDQwEHEnDhBkUWLsKv7bqbzOn3nEh/t/ARp/pBJtSbQybU+LGkAMhkGHAQ7Z2MVWzGinLKMvNqPJGdS+5G8RHYIJABJ0dHafUyCgii2aiVWWfhQ2Xz+KqOPjMLC7jrdy3ZnROFmaJa3Atdq0GsTmKsBedlNTgkkSs6k9iN5Q5jb2Wn3MbG3J2jgIBJ0nWiZ0dbNEmr1AAAgAElEQVSzNM2dxpMQUZc1l9fQ/+ICwlt8A4FHYO/Lh2sqr8fb8B835c2T1d9LFUheMU2BAhRZvJjkx48N3sckPVPe88QhtgN5ontz8f5FugV6c6lyNzj+I5xfZ8RSK8ZgZWVFeHi4CiZKtiKlJDw8/LlRbRmhmrZek5hjx7jVfwA21apSdPFihKVlpvI5cPkeHy4/Rbd3BWfifuBB3AMmJ9jS+vYV6LsHCnoaueRKZiUkJBAcHExcnFp0U8lerKyscHV1RaPRpDqu+kj0ZMdAAhC5aRMho8fg0LYtBad/m+bMYEOMWHeODWeDWd6vPL9em8rp0NN8EJvM53FmWAw4BDb5jFxyRVHeBqqP5A2Qp107nIZ8RtTmzdyfvyDT+YxvXZ78ua34avMt5jdcSI+yPfjN2oyPrZ4Qub4PJGd8VrCiKIqhVCB5zZw+/hiHjh24/9NPRK5fn6k8HKw1TO9YkWv3olmwP5AxNcYwtfZUztjY0C3+Gld2DzdyqRVFUZ5RgeQ1E0JQcPJkbOvUIWTSZKKP/J2pfBqUyU/XqkVYctiff289oH2p9qxo+TsJlrb0uruH3X9PM3LJFUVRtFQgyQaERkPhuXPIVaoUt4cOJc7PL1P5jGtdDhd7K4avO0dcQhIVnSuytv1mypCL4f7ezD0ykSTVzKUoipGpQJJNPN3HxMzenqABA0m4cyfDedhbaZje0RP/sBh+2Kvdv8Epd2F+ab+RzrGJLAvYyOC/BvEw3jTbbSqK8nZSgSQb0RQoQJEli0mOjSVo4ECSMrG/cr3SznSvXpQlRwI4c/OBNt88xZjY8mcmhEdyPOQ4PbZ1xz/S39jFVxTlLaUCSTZjVbo0rgvm8yTwJsFDMrePydj/laWQgzUjdE1cABSrTZc6E/jlzl2iY+7RY3sP9t3aZ+TSK4ryNlKBJBuyrVmTQl9/xePjxwmZMCHDM6FzW2mY0dGTgPsxzNp95dmJ6v2pXLYjawP9KZHLkWEHhvGTz08ky2QjP4GiKG8TFUiyKYe2bXEeOoSozVsImzcvw+nfLeXE+zWK8vPRG5wO1O5WhxDQ+gcK5Pfg12sXaOvaiIXnFjL0wFCi46ON/ASKorwtVCDJxhwHDcKhU0fCFy7iwbqMr5015n/lKORgzfB154iN1zVxaayh60pymVnw1eVjjKnyBUeCj9BjRw8CowKN+wCKorwVVCDJxoQQFJw0Cdt33+Xu5ClEHzmSofR2uSyY2cmTwPDHzNRv4spTFDr/irh/jR5+B1jadAmRcZF0396dw8GHjfwUiqLkdCqQZHNCo6HwnDnkKl2a20OHEefrm6H0tUs60atmMX795wYnb0Q8O+HeAJpMAd/NVPM/ytrWaymSuwiD9w1m9pnZPE54bNTnUBQl5zJpIBFCtBBCXBFCXBdCjE7jfB8hRJgQwkf36qd37jshxCUhhJ8QYp7QrWgohOguhLgghDgvhNglhHAy5TNkB+Z2thRZtAgzBwftPiYZnGMyumVZXPNaM2L9OR7HJz47UfszqNAB9k2l4F1fVrRcQYdSHfj14q+039yeQ0GHjPwkiqLkRCYLJEIIc+BHoCVQHuguhCifxqVrpZSVdK9lurS1gTqAJ+ABVAPqCyEsgLlAQymlJ3AeGGyqZ8hONAXyU3TJYpLj4jI8x8Q2lwXfdfTiZvhjvtul18QlBLRdAM7lYP1HWD8KZXLtySxvsRwbjQ2D9w9m2IFh3I25a4InUhQlpzBljaQ6cF1KGSCljAe8gbYGppWAFWAJ5AI0QCggdC9bXQ3FHsj4FPA3VK5SpXCdP087x+SzIRmaY1KrhCN9arux/J9AjgeEPzthaQvdVgISvHtC/GPeKfAOf7T+g2FVhnH09lHabGrDiksrSEhOMP5DKYryxntpIBFClBBC5NJ930AIMUQIkceAvAsDQXo/B+uO/VdHXTPVeiFEEQAp5THgABCie+2WUvpJKROAj4ELaANIeeDndMo9QAhxWghxOiwszIDivhlsa9ak0LSveXziBHfGj8/QHJORLcpQzNGGEevPEfNEr4krnzt0/BlCL8LWISAlGnMNfSv2ZVO7TVR3qc6s07Potq0bPvd8TPBUiqK8yQypkWwAkoQQJdF+aBcHVhuQLq1dmv77qbcVcNM1U/0FrADQ3asc4Io2+DQSQtQTQmjQBpLKQCG0TVtj0rq5lHKJlLKqlLKqs7OzAcV9czi0aYPzsKE83LKVsLlzDU5nY2nBzE5eBD+IZcauy6lPlmoKDcfBhXVwYBroAlRhu8LMbzSfOQ3nEPUkil47ezHl2BSinkQZ85EURXmDGRJIkqWUiUB7YI6U8nOgoAHpgoEiej+78p9mKClluJTyie7HpcA7uu/bA8ellNFSymhgJ1ATqKRL5y+1/xX/A6htQFlyHMeBA8nTuRPhixbz4I8/DE5XvXg++tR247djN/nH/37qk3W/hEo94fBM2DgIErX/NEIIGhdtzJZ2W+hdvjcbr22kzaY2bPHfovYfVxTFoECSIIToDvQGtumOaV5w/VOngFJCiOJCCEugG7BF/wIhhH5AagM8XT/9FrrOdV0tpL7u3G2gvBDiaRWjqV6at4oQApeJE7GtW5e7U6YSfdjw+R8jm5fFzdGGkevPp27iMjPTdr43HA/nveH39vD42ZBhG40Nw6sNTxkqPO7vcfTd05eAyABjPpqiKG8YQwLJh0AtYJqU8oYQojiw8mWJdLWYwcButB/2f0gpLwkhpgoh2uguG6Ib4nsOGAL00R1fD/ij7Qs5B5yTUm6VUt4BpgCHhRDn0dZQvjHwWXMcodFQ+IcfyFWmNMHDPif20iWD0llbmjOrsxe3I2P5dud/4rAQUH+Ets8k+BQsawLhqVcKLpOvDL+1/I1JtSZxJeIKHbd2ZN7ZecQlxhnr0RRFeYOIjDRNCCHyAkWklOdNVyTjq1q1qjx9+vTrLobJJITeI7B7N2RCAsW9vdEUTmtMw/O+3ubLsr9vsKpfDeqUTGM6zq3jsKa79vtuq6FYrecuCY8NZ/aZ2Wzx34KrnStja4ylrmvdrDyOoijZhBDijJSy6suuM2TU1kEhhL0QIh/a2sGvQojZxiikYhyaAvkpungxMu4JtzIwx2R48zK4O9kycv15HsWlMbS3aE3o9xfY5IPf2sD55/tiHK0dmfbuNH5p/gsacw2f7PuELw5+QWhMaFYfS1GUN4QhTVsOUsqHQAfgVynlO0AT0xZLySjtHJP5xN+8RfDgz0g2YI6JlcacmZ29CImK5Zsdl9O+yLEE9N0LRWrAn/3h4PSUEV36qrlUY8N7GxhSeQiHgw/TZlMbVvquJDE5MY1MFUXJSQwJJBa6TvEuPOtsV7Ih25o1KPTNNB6fPEnI2HEGjah6p1he+tV1Z83JWxy+ms58G5t80PNP8OoBB7+FjQNTRnTp05hr6O/Zn41tN1K5QGVmnJpBj+09uBB2IauPpihKNmZIIJmKtsPcX0p5SgjhDlwzbbGUzHJ47z2chw3j4bZthM0xbI7JF01L4+5sy+gN53mYVhMXgIUltPsJGo2H82ufG9Glr0juIixsvJDv639PeGw47+94n6+Pf632ileUHOqlgURKuU5K6Sml/Fj3c4CUsqPpi6ZkluPAAeTp3JnwxYt5sPblc0ysNNpRXHcfxvHN9heMphYC6j0d0XU6zRFdzy4VNHNrxuZ2m3m/3Pusu7qONhvbsD1gu5p7oig5jCGd7a5CiI1CiHtCiFAhxAYhhOurKJySOUIIXCZNxLZ+Pe5OnUr0oZev4lulaF7613PH+1QQa0/devGHfcVO0HsLxEXCssZw8590L7WztGNU9VF4t/KmoG1BRh8ZTf+9/dUmWoqSgxjStPUr2omEhdAuV7JVd0zJxoSFBa6zZ2NVpgzBn39h0ByTz5uUpkrRPIzacIFOi47hExSZ/sUpI7qc4Le2cG7tC/Mu51iOlf9byfga4/G970uHLR340edHniQ939eiKMqb5aXzSIQQPlLKSi87lp3l9HkkL5Jw7x6B3QyfY5KULFl3OohZe65yP/oJ7SoVYmSLshTKY512gtgHsLYXBB6B+qOhwWhtE9gL3I+9z8xTM9lxYwdFcxdlXI1x1C78Vq50oyjZmtHmkQD3hRA9hRDmuldPIPylqZRsQZM/P0WXLNHOMRkwkKSoFy+2aG4m6Fa9KAdHNOCTBiXYcfEujb4/yOw9V1Ivp/KUdV7tiK5K78Oh6emO6NLnZO3EjHozWNpsKUIIBv41kJGHRhL2OOes0qwobxNDaiRFgQVol0mRwD/AECnlLdMXzzje5hrJUzEnTnKrXz9sKlWiyM/LMLO0NChdUMRjZuy6zLbzIRSwz8WI5mXpULkwZmb/qXVICUe+h/1fQdHa0G2VdtjwSzxJesIvF39h2fllWJpb8lnlz+hapivmZuaZeUxFUYzI0BpJhpZI0ct8mJRyTqZK9hqoQKIVtXUbd0aMwL5VKwrN/A5hZvi+ZmduRjB1mx/ngiKpWNiBCa3LU714GoHi4gbY+DE4uML767QTGg1w8+FNph2fxrGQY5R3LM/EWhOp4FjB4PIpimJ8pg4kt6SURTNVstdABZJn7i9eQtgPP+A4YAD5v/g8Q2mTkyWbz93mu11XCImKo6WHC2NalqOoo03qC2+dAO/uIJOh6ypwq2NQ/lJKdgXu4rtT3xERF0G3Mt0YXHkwuS1zZ6iciqIYh6kDSZCUssjLr8weVCB5RkrJ3clTiFy7FpfJk8nbrWuG84iNT2LJ4QAWHfInKVny4btufNqwJPZWersLRATAqi7wIBDa/gheht/nUfwj5v87H+/L3jhZOzGy2kiauzVHvKQTX1EU41I1Ej0qkKQmExMJ+vRTYo78TZGFP2FXv36m8rkbFcfM3VfYcDYYR1tLvmhWmq5Vi2Bhrmsyy8SILn2X7l9iyrEp+EX4UbtQbcbVGEdR+zfm105R3nhZDiRCiEc8vzUuaLfQtZZSWmStiK+OCiTPS46J4WavD3gSGEix337D2iPz/REXgqP4apsvJwMjKFMgN+Nbl6NuKd3eY4nxsO1z8FkJFbtoN86yyGVw3knJSXhf8Wb+v/NJSEqgv2d/PvL4CEtzwwYLKIqSeSatkbxpVCBJW2JYGIFdu5GcEI/bGm8sXQ3bxyQtUkp2XbzLNzv9CIqIpVHZ/Iz9XzlK5rfTjuj6ezbsm5qhEV367j2+x3envmN34G7c7N0YX3M8NQrWyHR5FUV5ORVI9KhAkr4n168T2ON9LJydcVu9CnMHhyzlF5eQxPJ/Almw/zpxCUn0rFmMYU1KkcfGUm9EV2HosQ6cSmY4/6O3jzLtxDSCHgXRyr0Vw6sOx8k6jU25FEXJMhVI9KhA8mIxJ08S1Lcf1l5eFPnlZ4PnmLzI/egnzN57Fe+Tt8htpWFo41L0qlUMze1TmRrRpS8uMY5lF5bx88WfsTa3Ztg7w+hUuhNmwvDhzIqivJwKJHpUIHm5qG3buTN8OPb/+x+FZs3M0ByTF7l89yHTtvtx5Np93J1sGfu/cjR2eYxY3QUibmj7TLy6ZSrvgKgAph2fxsm7J/F08mRCrQmUzVfWKOVWFEUFklRUIDHM/aVLCft+NvatW+MyaSLmuY0zf0NKyYEr9/h6ux8BYTHUKenIxCaFKHPoU7hxGOqPggZjMjSiSz/v7Te2M/PUTCKfRNKjbA8GVx6MrcbWKGVXlLeZ0QJJOqO3ooDTwJdSyoBMl/IVUYHEMFJKwhctImzBj1gUyE+hb6djW6O60fJPSEpm1fGb/PDXNR7FJdDjHRfGsQzri6uhYmftfJMMjOjSF/Ukinln57Hu6jqcbZwZXX00TYo2UXNPFCULjBlIpgB3gNVoh/52A1yAK8DHUsoGWS6tialAkjGx585xZ+Qo4m/eJF+fPjh/PgyzXJn7gE9L5ON45u67xu/HbmKlMWOZ+xFq3lgARWtp+01sHTOd97mwc3x17CuuPLhC3cJ1GVtjLK651fY5ipIZxgwkJ6SUNf5z7LiUsqYQ4pyU0iuLZTU5FUgyLvnxY+7NmsWD1WvIVaokhWbMwKp8eaPewz8smm93+PGX3z0+yH2WSUnzMHMojHh/faZGdD2VmJzImstrWPDvApJkEgM9B9KnQh805pqXJ1YUJYUxl5FPFkJ0EUKY6V5d9M7l/A6Wt5SZjQ0uEydSZOlSkiKjuNG1G/cXLUYmprGUfCaVcLZjWe9qrOxbg5O29ekcO5aHUREkLm0EgX9nOl8LMwt6le/F5nabqedaj3n/zqPT1k6cunvKaGVXFOUZQ2ok7sBctMvIAxwDPgduA+9IKTP/F/+KqBpJ1iQ+eMDdqVN5tHMX1pUqUWjGdCyLFTPqPZKSJX+cDsJ792G+T5iGm9k9opvPJk+t3lnO+3DwYb458Q23o2/TpkQbvqz6JfmsMjYhUlHeRmrUlh4VSLJOSsnDbdu5+9VXyMRECowcSZ6uXYzemf0oLoFle32oeWootcwucbxIXzx7zsAmV9aapWITY1lyfgnLLy3HxsKGz9/5nA6lOqi5J4ryAsbsI3EF5gN10DZl/Q0MlVIGG6Ogr4IKJMaTcPcuIWPHEvPPMWzr16PgV1+hyZ/f6PcJCosk6PdB1H64k91mdYltOZc277g/v6FWBvlH+vPV8a84E3oGL2cvJtScQJl8ZYxUakXJWYzZR/IrsAUoBBQGtuqOKW8hjYsLRZYto8C4cTw+foIbbdrycPceo9+niHMean++hqAqI2mefITCW7vRa8EOTgdGZCnfEnlK8GvzX/m6ztfceniLrtu6MuvULGITY41UckV5+xhSI/GRUlZ62bHsTNVITONJQAB3Rowk7tIlHNq2ocD48UabxKgv+eJG5J8DuZOcl95PhlPO4x1GtyxLkXw2L0/8ApFxkcw5O4cN1zZQOX9lfmz8o9pES1H0GLNGcl8I0VMIYa579QTCs15E5U2Xy90dN+81OH3yCVHbthPQpi0xx08Y/T5mHu0x/3A7rjaJ7LCdyqPL+2k8+xAzdl3mUVxCpvPNY5WHybUnM7P+TC6EXaDv7r5ExGWtxqMobyNDAslHQBfgLhACdAI+NGWhlDeH0GhwHvIZbmtWY5YrF7f69CH02+kkP3li3BsVqYbovw+rPAVZoZnO5CLnWHjQn4azDrLm5C2SkjM/aKSFWwvmNppLQFQAfXb1ITQm1IgFV5Sc76WBREp5S0rZRkrpLKXML6VsB3R4BWVT3iDWnp4U3/gneXv0IGLFCm507Eicr69xb5LXDfruQRSrTY+QbzlR8xhu+WwY8+cFWs07wtHr9zOddT3XeixsspB7j+/Re1dvgh4FGa/cipLDZXbs4xdGLYWSI5hZW+MycQJFli0j+eEjbnTpyv1Fi4w6iRHrPNBzA1TuRQGf+azL/wsLu1Yg+kki7y87Qb8VpwgIi85U1tVcqrGs2TKiE6LpvbM31x9cN165FSUHy2wgMWgMphCihRDiihDiuhBidBrn+wghwoQQPrpXP71z3wkhLgkh/IQQ84RWbr1rfYQQ94UQczL5DIqJ2L1bB/ctm7Fv1oywOXO52bMX8TdvGu8G5hpoMx+aTEZcXE/LswP5a2AFRrYow/GACJr9cJgpWy8R+Tg+w1l7OHnwa/NfkUg+3P0hl8IvGa/cipJDZTaQvLRBWghhDvwItATKA92FEGkt1rRWSllJ91qmS1sb7bwVT8ADqAbUl1I+0ru2EnAT+DOTz6CYkHmePBSe/T2FZs3iSUAAAe3a88B7LUabACsEvPs5dF4BIT5YrWjGJx6SA8Mb0LmqKyv+CaTBrIMsP3qDhKTkDGVdKm8pVrRYgY2FDX139+VM6BnjlFlRcqh0A4kQ4pEQ4mEar0do55S8THXgupQyQEoZD3gDbQ0slwSsAEsgF6ABUvWACiFKAfmBIwbmqbwGDq1b4b5lMzaVK3N38mSCBg0i4d49492gQjvosx3iY2BZE5zvn+TbDp5sH1KXCoXsmbzVlxZzDrP/cmiGglhR+6KsaLkCZ2tnBu0dxN+3s/1KQIry2qQbSKSUuaWU9mm8ckspLQzIuzCg32MZrDv2Xx2FEOeFEOuFEEV09z4GHEA7SiwE2C2l9PtPuu5oazM5f42XN5x2EuPS1JMYd+023g1cq0K/v8CuAPzeHnxWU66gPSv71mDZB1WREj5afpoPfjnJlbuPDM7WxdaF5S2W4+bgxmf7P2NPoPEnXipKTmDKhYbS6kf574f+VsBNSukJ/AWsABBClATKAa5og08jIUS9/6TtBqxJ9+ZCDBBCnBZCnA4LC8vkIyjGIszMyNerJ8U3/onG1ZXbw4Zxe+RIkh4+NM4NdCO6KFYbNn0M+75CSEmT8gXYNaweE1qX51xQJC3nHmbsxgvcjzZseLKjtSM/N/8ZD0cPRhwewabrm4xTXkXJQUwZSIKBIno/u6LdICuFlDJcSvn0L3op8I7u+/bAcSlltJQyGtgJ1HyaTgjhBVhIKdNtvJZSLpFSVpVSVnV2ds760yhGkcvdHbc1q3H69FMebt9BQNt2xBw/bpzMn47oqvIBHJkFf/aDhDgsLczo+25xDo1oyAe13Fh7KoiGMw+y6JA/TxKTXpqtvaU9i5suprpLdSYcncAqv1XGKa+i5BCmDCSngFJCiOJCCEu0NYgt+hcIIQrq/dgGeNp8dQuoL4SwEEJogPp650DbrJVubUTJ3oRGg/Nng/UmMX5I6LffkhwXl/XMzTXw3jxoMgUuboDf2kCMdn5JXltLJrepwO5h9ahWPB/Td16myexD7LgQQnziizvkbTQ2LGi8gEZFGjH95HSWnF9ivIEDiknExifhe+chW8/dYe5f15i23Zc7kWpNNVMw6TLyQoj/AXMAc+AXKeU0IcRU4LSUcosQ4lu0ASQRiEC7de9l3Yivn4B6aJvDdkkpv9DLNwD4n5TysiHlUGttZV/JsbHcm/U9D1atwrJECQp9NwPrChWMk7nvZvhzAOR2gR7rwLl0qtNHroXx9TY/roQ+wtxM4JrXGjdHW4o7pX4VymONuW7V4cTkRCYcncC2gG18WOFDPn/nc7Uv/GskpeTeoyf434vG/34M/veiCdB9va0XNIQAcyGw0pgz5n9l6V6taJZXkn4bqP1I9KhAkv1F/32UkLFjSYyIwPnTT3Ds3x9hYciYjpcIPg1rukFSPHRdCcVTd7UlJiWzxzcUv5CHBNyPIfB+DDfux/A4/lmTl6W5GUUdbVICi5ujNf9ELuPQ3c10Lt2Z8TXHq31NTCwuIYmb4Y/xD4t+FizCogkIiyH6ybMJrzaW5rg721LC2Q53JztK5Nd+X9zJlnsPnzD6z/P84x9OLXdHZnT0pKhj1hb+zOlUINGjAsmbISkykrtTv+Lhjh1Ye3lpd2J0c8t6xg9uwuouEH5d2+xV+f0XXi6lJOzRk2eBJTyGG2ExBIbHEBj+WNcMJrF03kUup0PYxFejivUgSjg74OZkS3EnG4o72ZHXRqNqKxkgpeR+dHxKgPAPi075PujBY/Q/qgo5WFEivx3uTra6r9qg4WJv9cL3XEqJ96kgpm33IylZMqJ5GXrXdkupcSqpqUCiRwWSN0vU9u3cnTIVmZBAgVEjydO1a9Y/kGMjYV1vCDgIdb+EhuPBLOO1iKRkSUhULDd0QWZH0CouxnqjeeJB1M1uJCU9q0XZW1lQ3NmO4o7awOLmZIO77mtuq6zt+Pgmi09M5mZ4DP5hMc8FjUdxz2oXVhozijvZUeJpDUPvq41l1mqrIVGxjP3zAgeuhFGlaB6+6+RFyfx2WX20HEcFEj0qkLx5EkJDCRk7jpijR7GtV5eCX3+d9Z0YkxJg+5dwdgVU6ADtFoLGKstlXXN5Dd+c+IbqLjX4wvMbQqMkAboajDbgPOZOVGyq/1E72eXS1VxscXOyxV331c3RFiuNeZbLlB1ExMSnborSfb0V8TjVas0F7HOlChRPvy/kYG3SfgwpJZt8bjN5iy+xCUkMbVyKgfXcsTBXzZRPqUCiRwWSN5OUkgerV3Nv5izMcuXCZcpk7Fu0yGqm8M882DsRXKtD9zVg65Tlsm7x38KEoxOo6FSRHxv/iEMuh1Tnn7bx37gfk1KbuaFrNgt7lHpOSyEHK4o726bq+HdzsqVIXhssLbLXh1xCUjK3Ih4/q1Xo9V9EPn62V4ylhRnuTrapahVP+y5ed+3s3qM4Jm2+xM6Ld/EobM93Hb0oX8j+tZYpu1CBRI8KJG+2JwE3uDNqFHEXLmD/3nu4TBiPuX0W/9BfMqIrM/66+RcjDo+ghEMJFjddjKO1o0HpHsUlcDP8carO/qevqNhnH8ZPR5YV19Vc3PWCjf7IMlOIfByfZlPUrfDHJOrVLpzscmmbovT6L0o42VE4r2nLZww7LoQwcfNFIh8n8EnDkgxuWDLbBe5XTQUSPSqQvPlkQgL3Fy/h/sKFWDg7U+jbb7CtVStrmQaf0Y3oegLV+kOJhtpaioVlprM8evsoww4Mw8XWhaXNluJi65KlIj6IiU8JMIHhMS8cWVbM0SZVM9nT2kz+3LkM6mNKTEom+EFsqmDx9Gt4zLOVlDXmIiWQ6TdFuTvb4WD9Zvf9PIiJZ+o2Xzb+e5syBXLzXSdPvIrked3Fem1UINGjAknOEXvhAndGjiL+xg3yftCL/F98gZlVFvo5Im/B5k8h8CjIJNDYQLE62qDi3hDyl9NOQsiAs6Fn+XTfp+S2zM3SZkspZl8s8+VLx3Mjy/ReNyMep5pgaWNprq25ONtS3FEbZAo5WBESFZcqWNwMf0y83krJjraWzzVFlXC2wzWvdY7vR9h/OZSxf17k3qM4+td15/OmpXNM31VGqECiRwWSnOW5SYwzZmDtkcVJjHFREPi3dlSX/wEIv6Y9blcA3Btog4p7A7AvmG4W+nzDfRm0dxBmwowlzZZQOm/Wm84MlZQsub8gx3oAACAASURBVBMZm9LZr98vE/QgNlVHt4WZoKijzX+ChS3uTnbktc18zSwneBiXwLc7/FhzMojiTrbM6OhJ9eL5XnexXikVSPSoQJIzRR89SsjYcSSGh+P0ycc4DRhgnEmMAJFB2qDy9PVYt42vczltQCnRUFtzyZX+kNGAyAD67+lPXFIci5osoqJzReOULQsSkpIJinhMSFQcLg5WFM1ngyaH1y6y6uj1+4zacJ7gB7H0rlWMkS3KYpvLSL9n2ZwKJHpUIMm5kqKitJMYt2/HysuTQtOnk6t4cePeJDkZQi9CwAFtbeXWMUiMAzMNFKn+rMZSqDKYp/6ACX4UTP89/YmIi2BB4/+3d+fxUVX3/8dfn8lC1knCFvYdxAjIqqjsyBoLFhRi6/r7Vh7a2vb3dW1FK2pr3buptWJti1WWIq4g+yYKBRQMO2EnBEgi2ddZzvePOwlDiBAymZmEfJ6PBw9muXfmHELmPfeeez7nNQa1GlS3bVMBUVTm5KVl+/jXxiO0jY/k+Sl9GNLd96v96jsNEi8aJJe//CVLOPn0M5jychIffYT4lBT/zSp3lMLxTVaoHFoDJ1MBA03ioPNQzxHLKGjaBUTILM5kxvIZpBem8+qIVxnWruqKCKqh2HrkDI8uTOVQdhHTB7bn8eQrG/wFBheiQeJFg6RxOGcS41DPJMZEHycx1kTRd3B4neeIZS3kHbMej+sAXUdAlxHktOnLfV89zv4z+/n90N8zvrOP82FU0JQ6XPxxZRpvrT9Ii9gm/O7m3tyYlBjsZvmFBokXDZLGwxhDzty5ZL74kjWJcdZT2CdMCGQD4Myhs6fBDn8BZXmAUNC6Nw/YQ9hWfoZZ185kSs/pgWuXqnOp6bk8ujCVvacKmNy3DU/94CqaXmYXKGiQeNEgaXzKDh0m41e/ojQ1te4mMdaGywknt1eeBitJ38z/No/ny6hIHqEZd3b7oTW+ktirVrW/VHCVO928sfYAr60+QFxkGM9M7sXE3q0um2KdGiReNEgaJ+N0kv23v5H9Rh1OYvRVWSGOw+t4bPsfWVF2ip/m5HJfbj4S1Ry6DD97mXF8+4u9kqpH9pzM59GFqew4kce4qxJ59uZetIz1vY5bsGmQeNEgadxKduwk49FHrUmMd9xBy4d8nMRYB5xuJ7O+msXHBz/mzhbX8HBJCHJ4LRSetjZo1s0Kla4jodMQiIi74Oup4HO63Ly94TCvrthPZFgIv7kpiSn92zbooxMNEi8aJMpdUkLmK6+S8+9/E96lizWJsXev4LbJuHlxy4u8t+c9pnafypPXPkFI9n5rfOXQWmuCpKMYJATaDjg7277dQGtJYVUvHcwq5LGFqWw9msPwHi14bkpv2sZHBrtZtaJB4kWDRFXw6yTGWjDG8Nr213gr9S3GdxrPc0OeI6wiJJzlkL757GXGGdvAuCE8xjpKqThiad7jksu4KP9yuw1zNh7hhaX7CLFJg13eV4PEiwaJ8ubKy+PUs78l/7PP/DeJ8RL9Y+c/ePVra47JK8NfISK0mlNvJTnWVWAVV4TlHLYej21zdrZ9lxEQE4BLnlWNHD9TzGMfnF3e9/mpvenYLDrYzaoxDRIvGiSqOpWTGMvKSEhJIX76tKAGyoJ9C/jtpt8ysNVA/jLqL0SHXeQDJ+fI2dpgh9dZQQPWFWBdRlhHLB2vh3BdlzyYKpb3fW7xHhxuN4+M68ndDWR5Xw0SLxok6vs4TmeS+cIL5C9fDk4nUYMHk5CSQuzoUUhY4MchFh9azMwNM0lqlsRfb/zreQtkfS+3C06lnj0NdmwTuMohJBzaX3v2iKV1X7A1viq29cHJvBJmfriT1XszPcv79qFby9hgN+uCNEi8aJCoi3FmZZH7wSJyFyzAkZFBSIvmxE+dSsKttxLWtm1A27Lm2BoeXvcwHewdmD12Ns0ja1HTqbwYjn3lOWJZC6d3WI9HxFuXGXcdBb1vhfCGc5rlcmCM4ePtGcz6dBfFZS5+eWN3ZgzrUm8LZ2qQeNEgUTVlXC6KNmwgZ+48CtetAyBm2DDib0shZuhQJCQw3+Y3ndzEL1b/ghaRLZg9djZtYtr49oKFmXBonaea8RrIPwExrWDUTOj7Yz1KCbCsgjKe+mQnS3bU7+V9NUi8aJCo2nCcOEHOwoXkLlyIKyub0DatSZg2jfipUwlt0cLv7789czs/XfVTokKjmD12Np3j6mj8xhjr1NeKJyF9C7S8CsY+A91urJvXVzX2+Y6TPFmxvO+IrvxsVDeahNafUNcg8aJBonxhHA4KVq8hZ95cijdugtBQYkePJiFlOlHXXov4sbTJvjP7mLFiBgBvjXmLK5peUXcvbgzs/ghWzrIG7ruOgjHPQqvgzq9pbHKKynn2s90s2naCHokxvHjL1fStJ8v7apB40SBRdaXs8GFyF/yHvEWLcOXlEd6xI/HTpxP3w5sJTUjwy3sezjvMvcvvpdhZzBuj36Bvy751+wbOMtjyNqx70Vopsu+PrVNedh9Pp6lLsmZvJo9/uIPT+aX8ZGgXHqwHy/tqkHjRIFF1zV1WRsGyZeTMm0/JN98g4eHYJ4wnfnoKkf361nlZjIzCDO5dfi9ZJVn8edSfGdx6cJ2+PmBdPrz+Zdj8ljWb/voH4IZfQpP6fWXR5cRa3ncvczcfqxfL+2qQeNEgUf5Uum8/ufPnkffxJ7iLimjSowfxKdOJmzSJkJjvX4r3UmWXZHPv8ns5mn+UV4a/wsgOI+vstc+RcwRWPQM7P4DoFjDi19D/rvNWf1T+89WBbB5blMrxM8Fd3leDxIsGiQoEd1EReYsXkzNvHmW79yBRUcTddBMJKdOJSEqqk/fIK8vj/pX3s/u73fxuyO9I7pJcJ69brfStsPwJa2nh5lfAmGegxzgtxxIgxeXW8r7//OoIbeIieX5qb4Z29/9FHt40SLxokKhAMsZQunMnOXPnkb9kCaa0lIg+fUiYPh37xAnYIn0r4FfkKOLnq3/O1lNbeWLwE0y7YlodtbwaxsDexbDiN3DmIHQaCmN/C23qeJxGfa+vj57hkYWpHMoqYtrAdsxMTgrY8r4aJF40SFSwuPLyyPv4E3Lmz6f84EFssbHE3XwzCSnTadK1a61ft9RZykPrHmJ9+noeHPAg9/S6pw5bXQ2XA7b+A9Y9D8XfQZ/pMOpJXTclQEodLv60Ko231h+ieUx4wJb31SDxokGigs0YQ8nWreTMm2+VY3E4iBo0iPiU6cSOGYMt/NKXaHW4HTz+xeMsPbKUGX1m8EDfB/y/9kVpHmz4A2x8w7o/+H4Y+qCulxIgO9LzeGThtwFb3leDxIsGiapPnN99R96HH5IzfwGO48cJadqU+KlTiJ82jfD2l/YN3+V28eymZ/kg7QN+1PNHPHbNY9gkAOU2co/D6t9C6jyIagbDfwUD79F1UgKg3Onmr2sP8tqaNOwRYTw9+SqSe7f2y5cIDRIvGiSqPjJuN0VffkXO/HkUrl4DbjfRQ4aQkDKdmBEjarxOijGGl7e+zJzdc5jcdTKzrp9FqC1AV/hkbLcG5I98AU27wpinoedNOiAfAPtOFfDowm/5Nt2zvO/kXrS01+3Kn/UiSERkPPAnIAR42xjzfJXn7wZeAk54HnrNGPO257kXgWTABqwAfmmMMSISDrwGjADcwExjzAcXaocGiarvHKdOkbvwA3L/8x+cp08TmphI/K23En/rLYQlXvxcuDGGN1Pf5I3tbzCm4xheGPrC2QWy/M0YSFsOy5+E7H3Q4TprQL7dRT9/lI+cLjd/33CYV1bsJyLUxm9+cBVT63B536AHiYiEAPuBMUA6sAW4zRiz22ubu4GBxpgHqux7PVbADPM8tAH4tTFmrYg8DYQYY54QERvQ1BiTfaG2aJCohsI4nRSuXUvOvPkUbdgAISHEjBxBwvQUom+4/qLlWObsmsNLW1/ihrY38IcRfyAyNIBLvLqcsG0OrHkOirLgqilw41OQ0ClwbWik/LW8b30IkuuAWcaYcZ77vwYwxvzea5u7qT5IrsM66hgCCLAeuMMYs0dEjgM9jTFFNW2LBolqiMqPHyd3wQJyP1iE68wZwtq3J2H6NOKmTCG06ffPdl6UtohZX82iX8t+vDb6NWLDAzwzvawAvvwzfPUXMC64ZgYMfQiigjdDuzFwuw3vbjrKC0v3YpO6Wd63pkHiz1G5tsBxr/vpnseqmioiqSKyUETaAxhjNgJrgJOeP8s8IVJRyexZEflGRP4jIv6/Bk6pIAhv356WDz1Et7VraPPKy4QlJpL58iukDR/BiYcepnjLFqr7Ijil+xReHPYiqVmp/GT5T8gpzQlsw5vEWrW6fvEN9JkGG1+HP/ez/naWBbYtjYjNJtx1fSeW/f9hXN0+jpkf7uRHb28iq8D//+b+DJLqYrDq//pPgU7GmD7ASuBfACLSDbgSaIcVPqNEZBgQ6nnsS2NMf2Aj8HK1by4yQ0S2isjWrKysuuiPUkFhCw8nLjmZjv9+ly6ffUpCSgqF69dz9I47OXTTDzgz511c+fnn7DO+83j+NOpPHMw9yD1L7yGzODPwDbe3gcmvw30boG1/WPY4vH4N7Fxkjasov2jfNIp//8+1vDC1Nw6XwR7p/wsvgnpqq8r2IcAZY0yciDwCRBhjnvU89xugFGvcpBCINca4PUcwS40xV12oLXpqS11u3CUl5C/5nJz58ylNTUUiIrAnTyQhJYWIXr0qB1u3nNrCA6seoGlEU2aPnU272HbBa/SBlbD8N5C5C9oNsgbkO/ih+KSqZIzxaeC9Ppza2gJ0F5HOniutUoBPvDcQkdZedycBezy3jwHDRSRURMKA4cAeY6Xep1hXbAGMBnajVCNji4wkfuoUOi+YT6cPFhI3aRL5ny/lyK3TODL1FnIWLMBdVMSgVoN4e+zb5Jfnc9fnd3Eo91DwGt3tRrjvC5j0mjUP5Z1xMP92+O5g8Np0mfP7BNWK9/Hz5b8TgT9iXf77jjHmdyLyDLDVGPOJiPweK0CcwBngfmPMXs/RyRtYV20ZrKOOBz2v2RF4F4gHsoB7jDHHLtQOPSJRjYGroIC8Tz8ld+48ytLSsEVHEzd5EvHTUzjWEmYsn4HbuHlzzJskNaubIpK1Vl5kjZls+CO4ymDQT2DYoxDdLLjtUucI+lVb9YkGiWpMjDGUbNtO7vx55H++FFNeTmT//rgn38jP3XM5Ywp4ffTr9E/sH+ymQsFpWPscfDMHwmOtcivX3gdhdTuxTtWOBokXDRLVWDlzcsj78CNy5s/DcfQYEmdnbR8bn/UuZ+yQu0jpmULLqJbBbiZk7oEVT0HaMohrD6Ofgl5TwY/LGKuL0yDxokGiGjvjdlP83/+SM28+BatWgtPF/jbCxl4hxIwdwy3Xz6Bn057BbiYcWmeVXDmVCq37WgPynYcGu1WNlgaJFw0Spc5yZGaS99HHfPfpR7jTDuEW2NVByBjchf7TfsqQpPGBKfz4fdxu2LHAWqUx/wT0mGDV8GpxRfDa1EhpkHjRIFGqemUHDpD56UdkfbKIyJM5OG2Q1j2K6AnjGHLbg0THNQ9e4xwlsOmv8MWr4CiGAXdby/7GBHaVwMZMg8SLBolSF2aMoXBXKjvnvoms+pK4XAfloZDTvwtdb72HtmNuwhYRpAHwomxY+zxsfQfComDIL2HwzyA8KjjtaUQ0SLxokChVc26Xi9Q1C0j7zz9ot+U48cVQHhFK+IghtJ9yG9HXXYeEBWHdkew0WDkL9n4GsW1g9JPWSo22kMC3pT4yBkpyrNOBeSesvwtPW0dxtZxPokHiRYNEqdo5duYIKz56FcfytQzc4yC6DNz2aBImJBM3MZmogQOQkAB/kB/9CpbNhIxvILE3jH0Wuo4MbBsCzRgozYX8DE9IpJ9/Oz/DOgXozRYKjx6GCHut3laDxIsGiVK+yS/PZ9Gu+Wz/9J/02pbDoAMQ7jCEtGxB3IQJ2JOTiejdO2AzqXG7YdciWPU05B6DbmNgzDOQGOSJlrVVmnfhkMg7AY4qBc/FBrGtwd4W4tpaf1e9HdPSpyM2DRIvGiRK1Q2H28HKoyt5/5t3iNy8mxF7Q7j6oAub001Y+/bYJ07EPnEiEVf0CEyDnGWw+S1Y/5JVvr7f7TByJsS2Csz710RZgScM0q3TTefdPgHlBVV2EqsP1YZEO6sgZkwihPi3IKMGiRcNEqXqljGGbZnbeHf3u2zcv5LBafCDQ01psy8bXG6adO9WGSrhHTv6v0HFZ6ww2TzbWjf+hl/CdQ9Akxj/vm950dnxCO+xCe+QKMurspNYIWBvc24weN+ObWX1I8g0SLxokCjlP8fzj/Pe3vf4MO1DQvOKmJ7RgeF7bYTvtIoxRvTq5QmVCYS18vORwplDsPJp2P2R9WE9cqZ1lFKb0zvlxZ6xh2pCouJ2ae75+0W39ARDO8+RRJXbsa0hNNz3vgaABokXDRKl/C+/PJ9F+xfx3t73OFV0ij7utvy/01fQafNxyndbhb0jBw4gLjmZ2HHjLrjKo8+Ob7ZmyB//L7S40hqQ73bj2auXHKVnjxryT3hONWWcGxIlZ85/3ajmFw4JexsIbeK/fgWYBokXDRKlAqdiHGXOrjns/G4n9nA7d9vHMuZAFK7l6yg/eBBCQogePBh7cjKxN44mxF67q4ouyBjY84lVwyvnsFVyxbitwCjOPn/7yKZVxiSqhkTbBlFM0m3cpOWk8fXpr9mfs5+nrnuq1hdBaJB40SBRKvCMMWzP2s6cXXNYfXw1NrExvuM47ogYTvMNe8lfsgRHejoSFkb08GHETZxIzMiR2CIj67YhznJrMuOOBRDVrPqrm+xtGuwER4fbwd7v9vL16a/5+vTXfJP5Dfnl1oqZraJbseCmBSREJNTqtTVIvGiQKBVcxwuO8/6e91mUtohiZzGDWg3izivvYNCZBAo//5z8JZ/jzMpCoqKIHTkSe3Iy0UNuwBbeMMYSAqnMVcbO7J2VwbEtcxslzhIAOtk7MSBxQOWfNjFtfHovDRIvGiRK1Q9Vx1E62jty+5W384NOyfDtHvKXLKFg2TJcubnY7HZix9xIXHIyUddcg4T6f+3x+qjYUcz2rO2VwbEjawfl7nIAeiT0OCc4mkfWbW00DRIvGiRK1S9Ot9MaR9k9hx3ZO7CH27m1x63c1vM2WoY3pWjjRvIXL6Zg5SrcRUWENGuGffx47MkTiezbF7mM1ynJK8tjW+a2yuDY/d1uXMZFiISQ1CypMjT6texHXJM4v7ZFg8SLBolS9ZMxhm+zvmXO7jmsOrYKGzbGdx7PHUl3kNQsCXdpKYXr1pO/ZAmFa9diysoIbdMa+4QJxCUn0+TKKwM3m95PskuyK0Pj69Nfk5aThsEQZgujd/PeDEgcwMDEgVzd8mqiw6ID2jYNEi8aJErVf+kF6by3573KcZSBiQO5M+lOhrcfjk1suAoLKVy9mvzFSyj88ktwOgnv1Mmao3JTMk26dAl2F2okozDjnOA4kn8EgMjQSPq26Ft5xNG7RW+ahNTsUmLjdOIuKsJdVISrsBB3YRHuokLchYXEjh+vV23VBQ0SpRqOgvICFqUt4r0973Gy6GTlOMqkrpOICrOurHLm5FCwYgX5i5dQvHkzGEOTnj2xJ0/EPmEi4e3aBrkXFmMMR/OPVobG1tNbOVl0EoDYsBiuTejHwJgk+kR1p1N4K2xFpZUBcDYQinAXFuIuKsRV6LldWGiFRpG1jSkp+d42XLF9W62XANAg8aJBolTD43Q7WXnMmo9SdRwlMTqxcjtHZiYFS5eSv3gJJd9+C0Bk377WkcqE8YS2qNuFsCq//Rd6PtiLCivvOwsKyMw6QkbmQbKzj5GXcxpbSSmRZWB3hJLgakKMI4TwUhcUl4DLdfE3DA0lJCYGW8Wf6GhsMdGERHs9FhNNSHS053nrsZAY63545861rtCsQeJFg0Sphqu6cZRxncdxZ9KdJDU7t9pveXo6+Us+J3/JEsr27gWbjahrrsGePJHYESMAzjv1c87pIM83f+sxr2//hYW4PKeOLvTt31tpuOCOakJITCwRcU2JtDe1PtyjvT78vcMhuprHYmKQ8PCgjQNpkHjRIFHq8nCxcRRvZQcOWKGyeDHlR4/W7A3Cws5+s6/8pn/2g91ER5AlRRxzZXPQkcH+suPkhZZT0kRIaNqWHu2u5qqOg+jX8Tra2tv54V8gsDRIvGiQKHV5qTqO0iG2A7cn3c7krpMrx1EqGGMo3b2b4i1bsEVEVH7zt0VHn3vKKCbmvAmQxY5itmduZ+vprdYcjuwdONwOBKF7Qne/zuGoDzRIvGiQKHV5qhhHeXfXu6Rmp2IPt3NLj1u4redttIq+9ErD9WkOR32gQeJFg0Spy9/2zO3njaPckXQHVzW76nv3+b45HOG2cHq36F0ZHH1b9D3vSKcx0CDxokGiVOORXpDO+3utul5FjiIGJA6wxlHaDed08envncPRr2W/yuDo1bxXjedwXM40SLxokCjV+FSMo7y/530yijKICo2i2FkMQGx4LANanh3f6NmsJ2G24K9IWN9okHjRIFGq8XK6naw6toqvMr7iioQrGJA4gO4J3c+7ykudr6ZB0jjLaSqlGo1QWyjjOo1jXKdxwW7KZUsjWSmllE80SJRSSvlEg0QppZRPNEiUUkr5xK9BIiLjRWSfiBwQkV9V8/zdIpIlIts9f37i9dyLIrJLRPaIyJ/FU7VMRNZ6XrNin5b+7INSSqkL89tVWyISArwOjAHSgS0i8okxZneVTecbYx6osu/1wA1AH89DG4DhwFrP/R8bY/R6XqWUqgf8eURyDXDAGHPIGFMOzAMm13BfA0QA4UATIAw47ZdWKqWU8ok/g6QtcNzrfrrnsaqmikiqiCwUkfYAxpiNwBrgpOfPMmPMHq99/uE5rfVkxSkvpZRSweHPCYnVfcBXnUb/KTDXGFMmIvcB/wJGiUg34EqgoqD/ChEZZoxZj3Va64SIxAIfAHcAc857c5EZwAzP3UIR2VfLfjQHsmu5b0OlfW4cGlufG1t/wfc+d6zJRv4MknSgvdf9dkCG9wbGmO+87s4GXvDc/iGwyRhTCCAinwODgfXGmBOefQtE5H2sU2jnBYkx5i3gLV87ISJba1Ii4HKifW4cGlufG1t/IXB99ueprS1AdxHpLCLhQArwifcGItLa6+4koOL01TFguIiEikgY1kD7Hs/95p59w4CbgJ1+7INSSqmL8NsRiTHGKSIPAMuAEOAdY8wuEXkG2GqM+QT4hYhMApzAGeBuz+4LgVHADqzTYUuNMZ+KSDSwzBMiIcBKrCMZpZRSQdIoqv/6QkRmeE6TNRra58ahsfW5sfUXAtdnDRKllFI+0RIpSimlfKJB4lGDci73icgOz/yVDSKSFIx21qWL9dlru1tExIhIg77ixZeSPQ1VTX7GIjJNRHZ7ShK9H+g21rUa/Jz/4PUz3i8iucFoZ12qQZ87iMgaEdnmmbc3sU4bYIxp9H+wBu4PAl2wZtN/CyRV2cbudXsS1gUAQW+7P/vs2S4WWA9sAgYGu91+/hnfDbwW7LYGuM/dgW1Agud+y2C32999rrL9z7EuBAp62/38c34LuN9zOwk4Updt0CMSy0XLuRhj8r3uRnP+5MqGpqYlbJ4FXgRKA9k4P/ClZE9DVZM+3wu8bozJATDGZAa4jXXtUn/OtwFzA9Iy/6lJnw1g99yOo8qcPl9pkFhqVM5FRH4mIgexPlh/EaC2+ctF+ywi/YD2xpjPAtkwP6l1yZ4GrCZ97gH0EJEvRWSTiIwPWOv8o6Y/Z0SkI9AZWB2AdvlTTfo8C7hdRNKBJVhHYnVGg8RSk3IuGGNeN8Z0BR4DnvB7q/zrgn0WERvwB+ChgLXIv2pasqeTMaYP1hylf/m9Vf5Vkz6HYp3eGoH17fxtEYn3c7v8qUa/yx4pwEJjjMuP7QmEmvT5NuCfxph2wETgXc/veJ3QILFctJxLFfOAm/3aIv+7WJ9jgV7AWhE5glWi5pMGPOBeo5I9xpgyz93ZwIAAtc1favL/Oh342BjjMMYcBvZhBUtDdSm/yyk0/NNaULM+/w+wACqL4kZg1eGqExoklpqUc/H+5UoG0gLYPn+4YJ+NMXnGmObGmE7GmE5Yg+2TTMNdB8aXkj0N1UX7DHwEjATwlB/qARwKaCvrVk36jIhcASQAGwPcPn+oSZ+PAaMBRORKrCDJqqsG+LNoY4NhalbO5QERuRFwADnAXcFrse9q2OfLRg37+30lexqkGvZ5GTBWRHYDLuARc24x1QblEv5f3wbMM57LmBqyGvb5IWC2iPwv1mmvu+uy7zqzXSmllE/01JZSSimfaJAopZTyiQaJUkopn2iQKKWU8okGiVJKKZ9okCjlIxGZJSIP14N2HKlYilqpQNIgUUop5RMNEqWqISLRIrJYRL4VkZ0iMt37G7+IDBSRtV67XC0iq0UkTUTu9WzTWkTWe9a92CkiQz2P/1VEtnrW/3ja6z2PiMhzIrLR83x/EVkmIgdF5D7PNiM8r/mhZw2RN6urmSQit4vIZs97/01EQvz576UaNw0Spao3HsgwxlxtjOkFLL3I9n2wSudcB/xGRNoAPwKWGWP6AlcD2z3bzjTGDPTsM1xE+ni9znFjzHXAF8A/gVuw6pw947XNNVgzlXsDXYEp3g3xlMCYDtzgeW8X8ONL6LtSl0RLpChVvR3AyyLyAvCZMeYLkeqKrFb62BhTApSIyBqsD/stwDsiEgZ8ZIypCJJpIjID6/evNdZCQ6me5ypKeOwAYowxBUCBiJR6VeXdbIw5BCAic4EhwEKvtozGKji5xdPmSKChrzOi6jENEqWqYYzZLyIDsEpu/15ElmPV4Ko4io+ousv5L2HWi8gwrCOVd0XkJawjjYeBQcaYHBH5Z5XXqqg+7Pa6XXG/4vf1vPeqcl+Afxljfn2RCb+qxAAAAQRJREFUbipVJ/TUllLV8JyaKjbG/Bt4GegPHOFsafmpVXaZLCIRItIMa22PLZ6FkzKNMbOBv3teww4UAXkikghMqEXzrvFUerVhncLaUOX5VcAtItLS05emnrYo5Rd6RKJU9XoDL4mIG6vi8/1Yp4j+LiKPA/+tsv1mYDHQAXjWGJMhIncBj4iIAygE7jTGHBaRbcAurHLtX9aibRuB5z1tXA986P2kMWa3iDwBLPeEjQP4GXC0Fu+l1EVp9V+lGhARGQE8bIy5KdhtUaqCntpSSinlEz0iUUop5RM9IlFKKeUTDRKllFI+0SBRSinlEw0SpZRSPtEgUUop5RMNEqWUUj75P7jLR7DIzBFOAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x1659294f550>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# summarize results\n",
    "print(\"Best: %f using %s\" % (gsearch3_1.best_score_, gsearch3_1.best_params_))\n",
    "test_means = gsearch3_1.cv_results_[ 'mean_test_score' ]\n",
    "test_stds = gsearch3_1.cv_results_[ 'std_test_score' ]\n",
    "train_means = gsearch3_1.cv_results_[ 'mean_train_score' ]\n",
    "train_stds = gsearch3_1.cv_results_[ 'std_train_score' ]\n",
    "\n",
    "pd.DataFrame(gsearch3_1.cv_results_).to_csv('my_preds_subsampleh_colsample_bytree_1.csv')\n",
    "\n",
    "# plot results\n",
    "test_scores = np.array(test_means).reshape(len(colsample_bytree), len(subsample))\n",
    "train_scores = np.array(train_means).reshape(len(colsample_bytree), len(subsample))\n",
    "\n",
    "for i, value in enumerate(colsample_bytree):\n",
    "    pyplot.plot(subsample, -test_scores[i], label= 'test_colsample_bytree:'   + str(value))\n",
    "#for i, value in enumerate(min_child_weight):\n",
    "#    pyplot.plot(max_depth, train_scores[i], label= 'train_min_child_weight:'   + str(value))\n",
    "    \n",
    "pyplot.legend()\n",
    "pyplot.xlabel( 'subsample' )                                                                                                      \n",
    "pyplot.ylabel( 'Log Loss' )\n",
    "pyplot.savefig( 'subsample_vs_colsample_bytree1.png' )"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "colsample_bytree': 0.8, 'subsample': 0.7},\n",
    " -0.5852302830815046) 结果最优"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 对正则参数进行调优"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 88,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'reg_alpha': [1.5, 2], 'reg_lambda': [0.5, 1, 2]}"
      ]
     },
     "execution_count": 88,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#reg_alpha = [1e-3, 1e-2, 0.05, 0.1]    #default = 0\n",
    "#reg_lambda = [1e-3, 1e-2, 0.05, 0.1]   #default = 1\n",
    "\n",
    "reg_alpha = [ 1.5, 2]    #default = 0, 测试0.1,1，1.5，2\n",
    "reg_lambda = [0.5, 1, 2]      #default = 1，测试0.1， 0.5， 1，2\n",
    "\n",
    "param_test5_1 = dict(reg_alpha=reg_alpha, reg_lambda=reg_lambda)\n",
    "param_test5_1\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 90,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "D:\\anaconda\\lib\\site-packages\\sklearn\\model_selection\\_search.py:761: DeprecationWarning: The grid_scores_ attribute was deprecated in version 0.18 in favor of the more elaborate cv_results_ attribute. The grid_scores_ attribute will not be available from 0.20\n",
      "  DeprecationWarning)\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "([mean: -0.58474, std: 0.00356, params: {'reg_alpha': 1.5, 'reg_lambda': 0.5},\n",
       "  mean: -0.58454, std: 0.00350, params: {'reg_alpha': 1.5, 'reg_lambda': 1},\n",
       "  mean: -0.58500, std: 0.00361, params: {'reg_alpha': 1.5, 'reg_lambda': 2},\n",
       "  mean: -0.58494, std: 0.00349, params: {'reg_alpha': 2, 'reg_lambda': 0.5},\n",
       "  mean: -0.58443, std: 0.00371, params: {'reg_alpha': 2, 'reg_lambda': 1},\n",
       "  mean: -0.58516, std: 0.00339, params: {'reg_alpha': 2, 'reg_lambda': 2}],\n",
       " {'reg_alpha': 2, 'reg_lambda': 1},\n",
       " -0.58443300723981728)"
      ]
     },
     "execution_count": 90,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "xgb5_1 = XGBClassifier(\n",
    "        learning_rate =0.1,\n",
    "        n_estimators=340,  #第二轮参数调整得到的n_estimators最优值\n",
    "        max_depth=4,\n",
    "        min_child_weight=1,\n",
    "        gamma=0,\n",
    "        subsample=0.7,\n",
    "        colsample_bytree=0.8,\n",
    "        colsample_bylevel = 0.7,\n",
    "        objective= 'multi:softprob',\n",
    "        seed=3)\n",
    "\n",
    "\n",
    "gsearch5_1 = GridSearchCV(xgb5_1, param_grid = param_test5_1, scoring='neg_log_loss',n_jobs=-1, cv=kfold)\n",
    "gsearch5_1.fit(X_train , y_train)\n",
    "\n",
    "gsearch5_1.grid_scores_, gsearch5_1.best_params_,     gsearch5_1.best_score_"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 91,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "D:\\anaconda\\lib\\site-packages\\sklearn\\utils\\deprecation.py:122: FutureWarning: You are accessing a training score ('mean_train_score'), which will not be available by default any more in 0.21. If you need training scores, please set return_train_score=True\n",
      "  warnings.warn(*warn_args, **warn_kwargs)\n",
      "D:\\anaconda\\lib\\site-packages\\sklearn\\utils\\deprecation.py:122: FutureWarning: You are accessing a training score ('split0_train_score'), which will not be available by default any more in 0.21. If you need training scores, please set return_train_score=True\n",
      "  warnings.warn(*warn_args, **warn_kwargs)\n",
      "D:\\anaconda\\lib\\site-packages\\sklearn\\utils\\deprecation.py:122: FutureWarning: You are accessing a training score ('split1_train_score'), which will not be available by default any more in 0.21. If you need training scores, please set return_train_score=True\n",
      "  warnings.warn(*warn_args, **warn_kwargs)\n",
      "D:\\anaconda\\lib\\site-packages\\sklearn\\utils\\deprecation.py:122: FutureWarning: You are accessing a training score ('split2_train_score'), which will not be available by default any more in 0.21. If you need training scores, please set return_train_score=True\n",
      "  warnings.warn(*warn_args, **warn_kwargs)\n",
      "D:\\anaconda\\lib\\site-packages\\sklearn\\utils\\deprecation.py:122: FutureWarning: You are accessing a training score ('split3_train_score'), which will not be available by default any more in 0.21. If you need training scores, please set return_train_score=True\n",
      "  warnings.warn(*warn_args, **warn_kwargs)\n",
      "D:\\anaconda\\lib\\site-packages\\sklearn\\utils\\deprecation.py:122: FutureWarning: You are accessing a training score ('split4_train_score'), which will not be available by default any more in 0.21. If you need training scores, please set return_train_score=True\n",
      "  warnings.warn(*warn_args, **warn_kwargs)\n",
      "D:\\anaconda\\lib\\site-packages\\sklearn\\utils\\deprecation.py:122: FutureWarning: You are accessing a training score ('std_train_score'), which will not be available by default any more in 0.21. If you need training scores, please set return_train_score=True\n",
      "  warnings.warn(*warn_args, **warn_kwargs)\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "{'mean_fit_time': array([ 513.64428172,  552.62450156,  565.86558633,  427.75607719,\n",
       "         382.03448725,  317.18700953]),\n",
       " 'mean_score_time': array([ 1.74293399,  0.9316771 ,  0.82218599,  0.81095629,  0.7291389 ,\n",
       "         0.55307021]),\n",
       " 'mean_test_score': array([-0.58474301, -0.58453881, -0.58499851, -0.58493859, -0.58443301,\n",
       "        -0.58515793]),\n",
       " 'mean_train_score': array([-0.52414533, -0.52515542, -0.52646382, -0.52600731, -0.52672108,\n",
       "        -0.52791441]),\n",
       " 'param_reg_alpha': masked_array(data = [1.5 1.5 1.5 2 2 2],\n",
       "              mask = [False False False False False False],\n",
       "        fill_value = ?),\n",
       " 'param_reg_lambda': masked_array(data = [0.5 1 2 0.5 1 2],\n",
       "              mask = [False False False False False False],\n",
       "        fill_value = ?),\n",
       " 'params': [{'reg_alpha': 1.5, 'reg_lambda': 0.5},\n",
       "  {'reg_alpha': 1.5, 'reg_lambda': 1},\n",
       "  {'reg_alpha': 1.5, 'reg_lambda': 2},\n",
       "  {'reg_alpha': 2, 'reg_lambda': 0.5},\n",
       "  {'reg_alpha': 2, 'reg_lambda': 1},\n",
       "  {'reg_alpha': 2, 'reg_lambda': 2}],\n",
       " 'rank_test_score': array([3, 2, 5, 4, 1, 6]),\n",
       " 'split0_test_score': array([-0.57879931, -0.57861259, -0.57895173, -0.57906734, -0.57820354,\n",
       "        -0.57928317]),\n",
       " 'split0_train_score': array([-0.52582379, -0.52685414, -0.52820836, -0.52764522, -0.52869558,\n",
       "        -0.52940952]),\n",
       " 'split1_test_score': array([-0.58314705, -0.58284789, -0.58319029, -0.58350525, -0.58290159,\n",
       "        -0.58369115]),\n",
       " 'split1_train_score': array([-0.52324744, -0.52429904, -0.52563065, -0.52522478, -0.52572725,\n",
       "        -0.52708415]),\n",
       " 'split2_test_score': array([-0.58520569, -0.58560341, -0.58589242, -0.58530949, -0.58484581,\n",
       "        -0.58625875]),\n",
       " 'split2_train_score': array([-0.52407423, -0.52540083, -0.52678897, -0.52611779, -0.52674356,\n",
       "        -0.52791448]),\n",
       " 'split3_test_score': array([-0.5880548 , -0.58726571, -0.58799769, -0.58866107, -0.58764887,\n",
       "        -0.58813153]),\n",
       " 'split3_train_score': array([-0.52416132, -0.52502252, -0.52611822, -0.52562021, -0.52657781,\n",
       "        -0.52816272]),\n",
       " 'split4_test_score': array([-0.58850934, -0.58836559, -0.58896165, -0.58815079, -0.58856649,\n",
       "        -0.58842602]),\n",
       " 'split4_train_score': array([-0.52341989, -0.52420058, -0.5255729 , -0.52542853, -0.52586119,\n",
       "        -0.52700116]),\n",
       " 'std_fit_time': array([ 13.08654085,  18.63177443,   6.2177695 ,  65.93153443,\n",
       "         30.9598479 ,   6.6088887 ]),\n",
       " 'std_score_time': array([ 0.41478626,  0.1238598 ,  0.03790598,  0.05053858,  0.06821797,\n",
       "         0.05259363]),\n",
       " 'std_test_score': array([ 0.00355644,  0.00349835,  0.00361477,  0.00348824,  0.00370805,\n",
       "         0.00338819]),\n",
       " 'std_train_score': array([ 0.00091159,  0.00096007,  0.00097534,  0.00087088,  0.00106271,\n",
       "         0.00087411])}"
      ]
     },
     "execution_count": 91,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "gsearch5_1.cv_results_"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 92,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Best: -0.584433 using {'reg_alpha': 2, 'reg_lambda': 1}\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "D:\\anaconda\\lib\\site-packages\\sklearn\\utils\\deprecation.py:122: FutureWarning: You are accessing a training score ('mean_train_score'), which will not be available by default any more in 0.21. If you need training scores, please set return_train_score=True\n",
      "  warnings.warn(*warn_args, **warn_kwargs)\n",
      "D:\\anaconda\\lib\\site-packages\\sklearn\\utils\\deprecation.py:122: FutureWarning: You are accessing a training score ('std_train_score'), which will not be available by default any more in 0.21. If you need training scores, please set return_train_score=True\n",
      "  warnings.warn(*warn_args, **warn_kwargs)\n",
      "D:\\anaconda\\lib\\site-packages\\sklearn\\utils\\deprecation.py:122: FutureWarning: You are accessing a training score ('split0_train_score'), which will not be available by default any more in 0.21. If you need training scores, please set return_train_score=True\n",
      "  warnings.warn(*warn_args, **warn_kwargs)\n",
      "D:\\anaconda\\lib\\site-packages\\sklearn\\utils\\deprecation.py:122: FutureWarning: You are accessing a training score ('split1_train_score'), which will not be available by default any more in 0.21. If you need training scores, please set return_train_score=True\n",
      "  warnings.warn(*warn_args, **warn_kwargs)\n",
      "D:\\anaconda\\lib\\site-packages\\sklearn\\utils\\deprecation.py:122: FutureWarning: You are accessing a training score ('split2_train_score'), which will not be available by default any more in 0.21. If you need training scores, please set return_train_score=True\n",
      "  warnings.warn(*warn_args, **warn_kwargs)\n",
      "D:\\anaconda\\lib\\site-packages\\sklearn\\utils\\deprecation.py:122: FutureWarning: You are accessing a training score ('split3_train_score'), which will not be available by default any more in 0.21. If you need training scores, please set return_train_score=True\n",
      "  warnings.warn(*warn_args, **warn_kwargs)\n",
      "D:\\anaconda\\lib\\site-packages\\sklearn\\utils\\deprecation.py:122: FutureWarning: You are accessing a training score ('split4_train_score'), which will not be available by default any more in 0.21. If you need training scores, please set return_train_score=True\n",
      "  warnings.warn(*warn_args, **warn_kwargs)\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAZgAAAELCAYAAADkyZC4AAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMS4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvNQv5yAAAIABJREFUeJzs3Xd8jef/x/HXR4aE2Jsg9lZfYqsYpWjFbGvUKqVDB91Dq7p1aqtValYpNWOr1t7UCgmClJixQsjO5/fHffSXpsFJ5ORkXM/H4zyac849PofK+9zXdd3XJaqKYRiGYaS3XM4uwDAMw8ieTMAYhmEYDmECxjAMw3AIEzCGYRiGQ5iAMQzDMBzCBIxhGIbhECZgDMMwDIcwAWMYhmE4hAkYwzAMwyFcnV2AMxUtWlR9fHycXYZhGEaWsnv37ouqWuxu2+XogPHx8WHXrl3OLsMwDCNLEZG/7dnONJEZhmEYDmECxjAMw3AIEzCGYRiGQ+ToPpiUxMXFERYWRnR0tLNLMVLg4eGBt7c3bm5uzi7FMIy7MAGTTFhYGPny5cPHxwcRcXY5RhKqyqVLlwgLC6NChQrOLscwjLswTWTJREdHU6RIERMumZCIUKRIEXN1aRhZhAmYFJhwybzM341hZB0mYAzDMHISVdjyLdy87PBTmT4YwzCMnCIxEVa8Ajt/AgSaDXfo6cwVTA63bt06Hn744Xve5k6Cg4Np2rQpuXPn5vPPP7/tdgMHDqRChQrUq1ePevXqsXfv3jSf0zCMZBITYekLVrg0fwGaPuvwU5ormExOVVFVcuXKut8FChcuzDfffMOiRYvuuu1nn31Gz549M6Aqw8hBEhNg8bOwbza0fAVavwUZ0J9pAuYO3ltykENnrqXrMWuWzs+7nWvdcZvQ0FA6duxI69at2bp1Ky+++CITJkwgJiaGSpUqMXXqVLy8vFi+fDkjR46kaNGi1K9fn+PHj7N06dIUj7ljxw5efPFFoqKi8PT0ZOrUqVSrVu1f24wePZpjx45x+vRpTp06xauvvsqTTz4JQGRkJD179iQwMJAGDRowc+ZMRIQxY8awZMkSoqKiaNasGT/++ON/OuKLFy9O8eLFWbZs2T38yRmGkSYJcbBwGATOt4LF79UMO3XW/VqczR0+fJj+/fvz+++/M3nyZNasWcNff/2Fr68vX375JdHR0QwbNowVK1awadMmwsPD73i86tWrs2HDBvbs2cOYMWN48803U9xu//79LFu2jK1btzJmzBjOnDkDwJ49e/j66685dOgQx48fZ/PmzQAMHz6cnTt3EhgYSFRU1D8BN2HCBCZMmJDqz/3WW29Rt25dRowYQUxMTKr3NwwjifhYmPeEFS4PvJeh4QLmCuaO7nal4Ujly5enSZMmLF26lEOHDtG8eXMAYmNjadq0KcHBwVSsWPGfGw579+7NxIkTb3u8iIgIBgwYwNGjRxER4uLiUtyuS5cueHp64unpSevWrdmxYwcFCxakUaNGeHt7A1CvXj1CQ0Np0aIFa9euZezYsdy8eZPLly9Tq1YtOnfuzFNPPZXqz/zxxx9TsmRJYmNjGTp0KJ9++invvPNOqo9jGAYQHwNzB8CRFfDgx9D0mQwvwQRMJpU3b17A6oNp164ds2fP/tf7e/bsSdXxRo0aRevWrVm4cCGhoaG0atUqxe2SN2/dep47d+5/XnNxcSE+Pp7o6GieeeYZdu3aRdmyZRk9evQ93QRZqlSpf841aNCgOw4IMAzjDuKiYM7jELIGHvoCGg5xShmmiSyTa9KkCZs3byYkJASAmzdvcuTIEapXr87x48cJDQ0FYM6cOXc8TkREBGXKlAFg2rRpt91u8eLFREdHc+nSJdatW0fDhg1vu+2tMClatCiRkZHMmzcvFZ/sv86ePQtYobpo0SJq1659T8czjBwp9gbMegxC/gD/b50WLmACJtMrVqwY06ZNo3fv3tStW5cmTZoQHByMp6cn33//PR06dKBFixaUKFGCAgUK3PY4r776Km+88QbNmzcnISHhtts1atSIhx56iCZNmjBq1ChKly59220LFizIk08+SZ06dejateu/wihpH8y5c+fw9vbmyy+/5IMPPsDb25tr16zBE506dfqnn6dv377UqVOHOnXqcPHiRd5+++1U/VkZRo4Xcx1+eQRCN0LXH6B+f6eWI6rq1AKcydfXV5OvaBkUFESNGjWcVFHqREZG4uXlhary7LPPUqVKFUaMGJHm440ePRovLy9efvnldKwy/WWlvyPDyDDREVa4hO2C7hOhjuOG+4vIblX1vdt25gomC5s0aRL16tWjVq1aREREMGzYMGeXZBiGM0RdgRld4fRueGSqQ8MlNcwVTBa+gknJ1KlTGTdu3L9ea968OePHj3dSRekvq/8dGUa6unEJfu4K4cHw6Ayo1tHhp7T3Csaho8hEpAMwDnABflLVT5K9PxD4DDhte+k7Vf3J9t5Y4CGsq6zfgRdUVUVkHVAKiLLt015VL4hIS+BroC7QS1Xvrcc5ixo0aBCDBg1ydhmGYWSEyHCY0QUuhUCv2VDlAWdX9C8OCxgRcQHGA+2AMGCniASo6qFkm85R1eHJ9m0GNMcKC4BNgB+wzva8r6r++9IDTgIDgczdgWAYhpEerp+D6f5w9ST0nQsVWzm7ov9w5BVMIyBEVY8DiMivQBcgecCkRAEPwB0QwA04f8cdVENt50lMe8mGYRhZQMRpmN7ZCpnH54FPC2dXlCJHdvKXAU4leR5mey25HiKyX0TmiUhZAFXdCqwFztoeq1Q1KMk+U0Vkr4iMErMClWEYOcnVkzCtE9wIh34LM224gGMDJqVf/MlHFCwBfFS1LrAGmA4gIpWBGoA3Vii1sfWxgNU8Vge43/bol6qiRIaKyC4R2XW3+bsMwzAylcvHYWona9RYv0VQrrGzK7ojRwZMGFA2yXNv4EzSDVT1kqremtFwEtDA9nM3YJuqRqpqJLACaGLb57Ttv9eBWVhNcXZT1Ymq6quqvsWKFUvlR8p+MmI9mF9++YW6detSt25dmjVrxr59+9J8LMPIsS4ehakPQWwk9A8A7wZ338fJHBkwO4EqIlJBRNyBXkBA0g1EpFSSp/7ArWawk4CfiLiKiBtWB3+Q7XlR275uwMNAoAM/g9OpKomJWbtbqUKFCqxfv579+/czatQohg4d6uySDCNruRAM0x6ChFgYuAxK13N2RXZxWCe/qsaLyHBgFdYw5SmqelBExgC7VDUAeF5E/IF44DLWKDCAeUAb4ABWs9pKVV0iInmBVbZwccFqVpsEICINgYVAIaCziLynqvc2HfKK1+HcgXs6xH+UrAMdP7njJtltPZhmzZr983OTJk0ICwtLy5+cYeRM5wKtoci5XKxwKV7d2RXZzaF38qvqclWtqqqVVPVD22vv2MIFVX1DVWup6n2q2lpVg22vJ6jqMFWtoao1VXWk7fUbqtpAVeva9ntBVRNs7+1UVW9VzauqRe45XJwsu64HM3nyZDp2dPyNYIaRLZzZC9MfBhd3GLg8S4ULmOn67+wuVxqOlB3Xg1m7di2TJ09m06ZN9/rHYxjZX9humNkNcueHAUugcAVnV5RqJmAyqey2Hsz+/fsZMmQIK1asoEiRIqmq3TBynJPbYGZPyFvECpeC5ZxdUZqYyS4zueywHszJkyfp3r07P//8M1WrVr1jnYaR44Vugp+7Q74SVrNYFg0XMAGT6WWH9WDGjBnDpUuXeOaZZ6hXrx6+vnedI88wcqZja60rl4JlrQ79Aindm551mNmUs/BsymY9GMPIRo7+Dr/2hSKVof9i8Mq89+mZ9WByALMejGFkE8HL4dc+UKwaDFyaqcMlNcwVTBa+gkmJWQ/GMLKYQ4th3hNQ6j54fD54FnJ2RXeVKdaDyapU9T+jqbKK7L4eTE7+QmRkQwfmwYKh4O0LfeeBR35nV5SuTBNZMh4eHly6dMn8IsuEVJVLly7h4eHh7FIM497tnQULnoRyTeHxBdkuXMBcwfyHt7c3YWFhd70z3nAODw+Pf274NIwsa/d0WPICVPSzVqJ0z+PsihzCBEwybm5u/9wdbxiGke52TILlL0PlB+CxmeDm6eyKHMY0kRmGYWSUreOtcKnWCXrNytbhAiZgDMMwMsamr2DVm1DDHx6ZDq65775PFmeayAzDMBxt/VhY+yHU7gndfgSXnPGrN2d8SsMwDGdQhT8/gI2fw329oct4a12XHMIEjGEYhiOowu+jYMu3UL8/PDwOcuWsXgkTMIZhGOlNFVa+DtsnQMMh0PGzHBcuYALGMAwjfSUmwvKXYNcUaPIsPPghZNGZQe6VCRjDMIz0kpgAS56HPTOhxQho+26ODRdw8DBlEekgIodFJEREXk/h/YEiEi4ie22PIUneGysiB0UkSES+EdvkYCKyznbMW/sUt72eW0Tm2M61XUR8HPnZDMMw/iUhHhY9bYWL32s5PlzAgVcwIuICjAfaAWHAThEJUNVDyTado6rDk+3bDGgO1LW9tAnwA9bZnvdV1X9PgwyDgSuqWllEegGfAo+l1+cxDMO4rYQ4a9LKgwugzdvQ8hVnV5QpOPIKphEQoqrHVTUW+BXoYue+CngA7kBuwA04f5d9ugDTbT/PA9pKVp0S2TCMrCM+Fn4baIVLu/ezRLhci47LkAl9HRkwZYBTSZ6H2V5LroeI7BeReSJSFkBVtwJrgbO2xypVDUqyz1Rb89ioJCHyz/lUNR6IAIokP5mIDBWRXSKyy0xoaRjGPYmLhrn9IHgpdPgUmj/v7IruSFVZ8FcYrT9bR8C+Mw4/nyMDJqWrh+SRuQTwUdW6wBpsVyAiUhmoAXhjBUcbEWlp26evqtYB7rc9+qXifKjqRFX1VVXfYsWyx6pxhmE4QVwU/NobjqyEh76EJk85u6I7CrkQSe9J2xg5dx/liuShaol8Dj+nI0eRhQFlkzz3Bv4Vmap6KcnTSVj9JgDdgG2qGgkgIiuAJsAGVT1t2/e6iMzCaoqbkeR8YSLiChQALqf3hzIMwyD2Bsx6DEI3gf93UL/f3fdxkui4BMavDWHC+mN4urnwUbc69GpYlly5HN+D4MgrmJ1AFRGpICLuQC8gIOkGIlIqyVN/4FYz2EnAT0RcRcQNq4M/yPa8qG1fN+BhINC2TwAwwPZzT+BPNauGGYaR3mKuw8ye8Pdma16xTBwu64+E0/6rDXz7Zwid65bmj5da0adxuQwJF3DgFYyqxovIcGAV4AJMUdWDIjIG2KWqAcDzIuIPxGNdbQy07T4PaAMcwGrmWqmqS0QkL7DKFi4uWM1qk2z7TAZ+FpEQ27F6OeqzGYaRQ0VHWOFyejf0+Alq93B2RSm6cC2aMUsPsXT/WSoWzcusIY1pVrlohtchOflLvq+vr+7alXy0s2EYRgpuXoaZ3eFcIDwyFWp0dnZF/5GQqMzc9jefrzpMTEIiw1tXZphfRXK7pu8EmyKyW1V977aduZPfMAzjbm5cgp+7QPhhaxXKah2cXdF/HAiL4M2FBzhwOoL7qxTl/S618Sma16k1mYAxDMO4k8gLMN0frpyA3rOtpY4zkevRcXyx+ggztoZSxCs33/b+Hw/XLUVmuA3QBIxhGMbtXDsLM/whIgz6zIWKfs6u6B+qyvID53hvyUHCI2Po16Q8L7WvRgFPN2eX9g8TMIZhGCmJCIPpna0rmMfnQ/lmzq7oHycv3WTU4kDWHwmnVun8TOrvy31lCzq7rP8wAWMYhpHclb+tcIm6Av0WQtlGzq4IgJj4BCZtOM63f4bg5pKLdzvXpF+T8ri6ZM61ZkzAGIZhJHXpmNXnEnsd+i+GMvWdXREA245f4q2FBzgWfoOH6pRi1MM1KVnAw9ll3ZEJGMMwjFsuHrWuXOJjYMBSKFX37vs42KXIGD5aHsz8v8IoW9iTqQMb0rp6cWeXZRcTMIZhGAAXgqwrFxQGLoMSNZ1aTmKiMnfXKT5eEczN2HiebV2J4a2r4Omevve0OJIJGMMwjHMHYEYXyOUGA5ZAsapOLSf43DXeWhjI7r+v0KhCYT7sWpsqGTA5ZXozAWMYRs52Zg/M6Aruea1wKVLJaaXcjI1n3B9HmbzxBPk8XPn8kfvoUb9MprinJS1MwKTFqR2wayp0+Q5yZZ3LVcMwkjm1E2b2AI8CMHAJFPJxWilrDp3n3YCDnL4axWO+ZXm9Y3UK5XV3Wj3pwQRMWlw8AvtmQeGK4Jf5V68zDCMFf2+FXx6BvEWtK5eCZe++jwOcuRrF6ICDrD50nqolvPjtqaY09CnslFrSmwmYtKjXF46vg3UfgU/zTHUDlmEYdjix0VrPJX9pGBBg/TeDxSUkMm1zKF+tOUKiKq93rM7gFhVwy6T3tKSFCZi0ELFWsAvbBfOHwFObIE/2+MZhGNnesT9hdh8oVB76B0C+Ehlewl8nr/DmggMEn7tO2+rFGe1fi7KF82R4HY6WfaIyo3nkt6bsjrwAi5+FHLzsgWFkGUdWw6xeVkf+wGUZHi4RN+N4c+EBevywhYioOCY83oCfBvhmy3ABcwVzb0r/D9qNgVVvwI6J0HiYsysyDON2gpbCbwOt+1v6LcrQVgdVZdHe03y4LIgrN+MY3LwCL7arilfu7P0rOHt/uozQ5Gk4sR5Wvw1lG0Ppes6uyDCM5A4utJqzS9WzJq70zLiJIY+FR/L2wkC2Hr9EvbIFmf5EbWqVLpBh53cm00R2r0Sgy/eQpyjMe8Jar9swjMxj/2/Wv03vhtbElRkULtFxCXz5+xE6fr2RwDMRfNC1NguebpZjwgUcHDAi0kFEDotIiIi8nsL7A0UkXET22h5Dkrw3VkQOikiQiHwjye40EpEAEQlM8vw+EdkqIgdEZImI5HfkZ/uXvEWs9bmvnIBlL5n+GMPILPb8AguehPLNoe88q+80A2w4Es6DX2/gmz+O0qlOSf58qRWPNylPrlxZ84bJtHJYwIiICzAe6AjUBHqLSEqT+8xR1Xq2x0+2fZsBzYG6QG2gIfDPSj8i0h2ITHacn4DXVbUOsBDI2BtUfJqD3+uwfw7sm52hpzYMIwW7psLiZ6BiK2uxsNxeDj/lhWvRPDd7D/2n7CCXCL8MaczXvf5HsXy5HX7uzMiRfTCNgBBVPQ4gIr8CXYBDduyrgAfgDgjgBpy3HccLGAkMBeYm2acasMH28+/AKmDUPX+K1Gj5MoRutK5ivBtC0SoZenrDMGy2T4QVr0CV9vDoz+Dm2GntExKVX7b/zWcrDxOTkMiIB6oyzK8iHm45e6YPRzaRlQFOJXkeZnstuR4isl9E5olIWQBV3QqsBc7aHqtUNci2/fvAF8DNZMcJBPxtPz8CZPxtublcoPskcPO0RqvERWd4CYaR4235zgqXag/BYzMdHi6BpyPo/v1m3ll8kPvKFmTViy154YEqOT5cwLEBk1JjY/LOiSWAj6rWBdYA0wFEpDJQA/DGCqU2ItJSROoBlVV1YQrHfgJ4VkR2A/mA2BSLEhkqIrtEZFd4eHhaPted5S8FXX+A84HWyDLDMDLOxi9g9VtQsys8Oh1cHdc0dT06jveWHMT/u02cvhrNuF71+HlwIyoUzeuwc2Y1jmwiC+PfVxHewJmkG6jqpSRPJwGf2n7uBmxT1UgAEVkBNAGuAw1EJBSr9uIisk5VW6lqMNDetn1V4KGUilLVicBEAF9fX8f0xld9EJoOh63fQYWWUNP/7vsYhpF2qrD+U1j3MdR51PqS5+KYX2+qyorAc7y35CAXrsfweOPyvPxgNQp4ujnkfFmZI69gdgJVRKSCiLgDvYCApBuISKkkT/2BW81gJwE/EXEVETesDv4gVf1BVUurqg/QAjiiqq1sxypu+28u4G1ggsM+mT3avmvdiBkwHK6edGophpGtqcIfY6xwqdcXuk1wWLicvHSTQdN28swvf1Ekb24WPtOc97vWNuFyGw4LGFWNB4ZjdbYHAXNV9aCIjBGRW1/pn7cNRd4HPA8MtL0+DzgGHAD2AftUdcldTtlbRI4AwVhXSlPT9QOllqs79JwCiYkwbzAkxDm1HMPIllStpuhNX0KDgeDvmCU0YuMTGb82hHZfrWfnicu883BNAoY3p17ZjLthMysSzcH3bPj6+uquXbsce5LA+dZNXi1GwgPvOvZchpGTqMKK12DHj9BoKHQca934nM62H7/EW4sCCbkQScfaJXm3cy1KFnDswIHMTkR2q6rv3bYzU8U4Wu0e1tT+m76CCvdDpTbOrsgwsr7ERFg2AnZPs/o723+Q7uFy+UYsHy0PYt7uMLwLeTJloC9tqmf8zMtZWaoCxta/4aWq1xxUT/bU4VNrFcwFw+DpzeBV3NkVGUbWlZgAAc/B3l+sloG276RruCQmKr/tPsXHK4KJjI7nmVaVeK5NFTzdzbDj1LprH4yIzBKR/CKSF+smycMiYpZxTA33PNBzKsRcg4XDrG9fhmGkXkK89W9o7y/Q6o10D5fD567z2MStvDb/AFWL52P5C/fzaofqJlzSyJ5O/pq2K5auwHKgHNDPoVVlRyVqQodPrMWOtoxzdjWGkfUkxMH8wXDgNytYWr2ebuFyMzaeT1YE89A3Gwm5EMlnPesyZ1gTqpbIly7Hz6nsaSJzsw0V7gp8p6pxIpJzRwbciwYDran9/3jfmnyvbCNnV2QYWUN8jDVYJngptP8Qmg1Pt0P/EXSedxYf5PTVKB719eaNjjUolNc93Y6fk9kTMD8CoVjDhTeISHnA9MGkhQh0Hgend1tDl5/aAJ6FnF2VYWRucdEwtx8cXQ0dP4PGQ9PlsGcjohgdcJBVB89TpbgXc4c1pVEFs/R5ekrTMGURcbXd55KlZcgw5ZSE7YYp7aFaJ3h0hkOGVhpGthB7E37tA8fXwsNfg++gez5kfEIi07aE8tXvR0hQ5YW2VRncogLurmZ5LHvZO0zZnk7+F2yd/CIik0XkL8CMtb0X3g2sO/2DAmDXZGdXYxiZU0wkzHrUGubfZXy6hMuek1fo/N1mPlgWROOKRfh9hB9Pt6pkwsVB7Gkie0JVx4nIg0AxYBDWXfKrHVpZdtd0OJzYACvfhLJNoGRtZ1dkGJlH9DUrXE5th+4Toe6j93S4iJtxjF0VzKwdJymRz4MJj9fnwVolEdN64FD2xPatv4FOwFRV3UfKMyUbqZErlzUhn2chmDcIYm84uyLDyByirsLP3SBspzXd0j2Ei6qyaM9p2n65jtk7TvJE8wqsecmPDrVLmXDJAPYEzG4RWY0VMKtEJB9gbuRID17FrG9nF4/C8ledXY1hON/NyzCjC5zdZ/VP1uqW5kMdD4/k8cnbeXHOXsoUykPA8BaMergmXrnNBCYZxZ4/6cFAPeC4qt4UkSJYzWRGeqjoZ62EueEz6+d7bAowjCzrxkUrXC4ehV6zoGr7NB0mOi6BH9Yd44d1x8jtlosPutamd6NyuOQyVywZ7a4Bo6qJIuIN9LFdUq63Y2ZjIzX8XofQTbB0BJRpAEUqObsiw8hY18/DDH+4Egq9Z0Pltmk6zMaj4YxaFEjopZt0qVeatx6qQfF8OXtiSmeyZxTZJ8ALWNPEHMKaYv9jRxeWo7i4Qo+fwMXN6o+Jj3F2RYaRca6dgWkPWesm9f0tTeFy4Xo0z8/eQ7/JOxARZg5uzLhe/zPh4mT2NJF1AuqpaiKAiEwH9gBvOLKwHKeAN3T5Hn7tDb+/Cx0/cXZFhuF4V0/B9M5W89jjC6B801TtnpCozNpxkrErg4mJS+TFB6rwlF8lPNzM3GGZgb29XQWBy7afCzioFqN6J2j8FGz/wVpquXonZ1dkGI5zJdQKl6gI6LcQyjZM1e6BpyN4a1Eg+05dpXnlIrzfpTYVi3k5plYjTewJmI+BPSKyFmt4ckvM1YvjtBsDf2+Bxc9Aqc1QoIyzKzKM9HfpmBUusTdgwGJreXE7RcbE8+XqI0zbcoLCed0Z16se/veVNsOOMyF7Ovlni8g6oCFWwLyGA5dazvFcc8Mj0+DHljB/CAxY4rD1xQ3DKcKPWOGSGAcDl0LJOnbtpqqsDDzHe0sOcf56NH0bl+OVB6tTwNPNwQUbaWVXUKjqWVUNUNXFqnoO2GbPfiLSQUQOi0iIiLyewvsDRSRcRPbaHkOSvDdWRA6KSJCIfCPJvp6ISICIBCZ5Xk9EttmOs0tEsu5UxUUqwUNfwsktsGGss6sxjPRz/hBM6wSaCAOX2R0upy7f5IlpO3n6l78onNedBU8344OudUy4ZHJp/Wp812tREXEBxgPtgDBgp4gEqOqhZJvOUdXhyfZtBjQH6tpe2gT4Aets73cHIpMdZyzwnqquEJFOtuetUvGZMpf7HrOm9l8/FnxaWH0yhpGVnd1v3efimtu6Mi9a5a67xMYn8tOm43zzx1FcRBj1cE0GNC2Pq4tpRMkK0how9kzB3AgIUdXjACLyK9AFa6izPcf3ANyxwswNOG87jhcwEhgKzE22T37bzwWAM3acJ3PrONZaann+k9ZSy3mLOrsiw0ib039Z07+4e8GAALvu9dpx4jJvLTzA0QuRdKhVknf9a1KqgGcGFGukl9sGjIh8S8pBIlijyu6mDHAqyfMwoHEK2/UQkZbAEWCEqp5S1a22QQVnbef7TlWDbNu/D3wB3Ex2nBexprL5HKvpr5kdNWZuub3gkakwqS0sfAr6zLXmMDOMrOTUDpjZAzwLwoClUKj8HTe/fCOWj5cH8dvuMMoU9GTyAF/a1iiRQcUa6elOVzB3WijFnkVUUmpGSx5YS4DZqhojIk8B04E2IlIZqAF427b73RZC14DKqjpCRHySHetprICaLyKPApOBB/5TlMhQrKsfypUrZ8fHcLKSdeDBD2H5y7BtPDR7ztkVGYb9/t4CvzwCXsWtZrEC3rfdNDFRmfdXGB8vD+J6dDxPt6rE822q4Olu7mnJqtK04JhdBxZpCoxW1Qdtz98AUNUUZwGw9dlcVtUCIvIK4KGq79veeweIBq4Do4BYrHAsDmxR1VYiEgEUVFW1DQiIUNX8KZ3rFqctOJZaqjDncTiyEgavtqaTMYzM7vh6mN3LCpX+AZC/1G03PXL+Om8vDGRH6GUa+hTiw251qFoiXwYWa6RGui2/rTDgAAAgAElEQVQ4dg92AlVEpIKIuAO9gICkG4hI0v/j/IFbzWAnAT8RcRURN6wO/iBV/UFVS6uqD9ACOKKqrWz7nLFtB9aCaEcd8JmcQwS6fAf5SlnrkkdHOLsiw7izkDXWei6FfKzRYrcJl6jYBD5dGUyncRs5cuE6Y3vUZc7QpiZcsgmH3WChqvEiMhxYBbgAU1T1oIiMAXapagDWvGb+QDzWTAEDbbvPwwqJA1jNaivtmGDzSWCciLhiXe2kz8LdmYVnIegxGaZ2hCUvWutkmBvLjMzo8EqY2w+KVYN+iyFvkRQ3+zP4PO8sPkjYlSgeaeDNG51qUDivewYXaziSw5rIsoIs00SW1MYv4I8x0PkbaDDA2dUYxr8FLYHfBlkrtD6+APIU/s8mZyOieC/gECsPnqNKcS8+6FqbxhVTDiEjc7K3ieyuVzAi8k0KL0dgXYUsTktxxj1oPsJaannFa1C2ERSv4eyKDMMSuMCafaJMfXh8Pnj8e9rC+IREpm/9my9XHyZBlVc7VGNIi4q4u5qRkdmVPX+zHlgLjh21PeoChYHBIvK1A2szUpIrF3SbaA1h/m0QxCYfrW0YTrBvDswfDGUbWxNXJguXvaeu4v/dZt5feoiGFQrz+wg/nmlV2YRLNmdPH0xloI2qxgOIyA/Aaqw79A84sDbjdvKVgG4/wszusOoN6DzO2RUZOdlfP0PAc9aME33mgHvef96KiIrjs1XB/LL9JMXz5eaHvvXpULukmZgyh7AnYMoAebGaxbD9XFpVE0TErIzlLJXbQosRsOkrqOAHtbs7uyIjJ9o5GZaNhEpt4LFfwD0PYE1MGbDvDO8vDeLyjRgGNavAyPZV8cptJm7NSez52x4L7LXNqHxruv6PRCQvsMaBtRl30/otCN0MS16wpjsvXMHZFRk5ybYJsPI1qPIgPDoD3KzVI09cvMGoRYFsCrnIfd4FmDaoIbXLmGWkciK7RpHZ7ldphBUwO1Q168/zRRYdRZbclb/hx/uhcCV4YhW4mmGeRgbY/A38PgqqPww9p4KrO9FxCUxYf4zv1x0jt0suXu1QjT6Ny+OSyzSHZTfpNorMpiFwv+3nBLLDRJLZRaHy4P8tzO0Pf46B9h84uyIju9vwGfz5AdTqDt0ngosbm45eZNTiQE5cvIH/faV5++EaFM/n4exKDSezZ5jyJ1gB84vtpedFpJmqmlUtM4uaXcB3MGz51uqPqdLO2RUZ2ZEqrPsY1n8KdR+DLt8TfjOBD5btYfHeM/gUycPPgxtxf5Vizq7UyCTu2kQmIvuBeqqaaHvuAuxR1bp33DELyBZNZLfERcNPbeH6WXhq8x3nfTKMVFOFNaNh89dQ73ESHx7HrF2n+XRlMDFxiTzdqhJPt6qEh5uZmDInSO+5yJJOz5/je+suRsaw6ehFZ5fxb24eVlt4XBQseBISE5xdkZFdqMKqt6xw8X2Cgw0/oPuP23l7USB1yhRgxYv3M6JdVRMuxn/YEzAfA3tEZJqITAd2Ax85tqzMbdrmUB6fvJ0BU3YQdPaas8v5f8WqQqfPIXQjbPzS2dUY2UFiIix/BbaNJ9Z3KO/rEDp/t4WwKzf5+rF6/DKkMZWKeTm7SiOTSs0osoZYo8i2A7myw0iytDaRxcQn8PPWv/n2zxCuRcfRs743I9tXzRyr7anCgqEQOM+axbZ81l93zXCSxERY+gL8NYPjVQfTN7QT567H0KdROV59sDoF8rg5u0LDSextIkvTZJciclJVs8BqXXd2r30wETfjGL8uhGmbQ8mVCwa3qMBTfpXI5+Hkf3gx1+HHlhAfA09tSnHCQcO4o8QEWDwc9s1iaYE+DD//EDVKFeDDbrWpX66Qs6sznMzRAXNKVcumqbJMJL06+U9dvskXqw+zaO8ZCud154W2VejdqJxz51k6sxd+esAaUdZrlpna37BfQjyJC4aS6+B8xiU+wo/0ZGS7qgxs5oOri5k7zHD8gmM5d47/FJQtnIeve/2PJcNbUK1EPt4NOEj7r9az4sBZnLYcQul60P59OLwctv/onBqMrCchjsszHifXwfl8EteLQ1WeYs1IP4bcX9GEi5Fqt72CEZFvSTlIBBhwt+WIswJHDFNWVdYdDufjFUEcOR9J/XIFebNTDXx9nNBMpQqze8OxP2Dw71boGMZtXIm4zpmfelHr+ia+cR1Eze5v8EDNEs4uy8iE7rmJTETuuJqVqk5PY22ZhiPvg4lPSGT+X2F8sfoIF67H8GCtErzWoToVM3rEzc3L8ENzaxjzsA2Q2yxFa/ybqrJgRwglVjxJC/awuvzLtOj7BnnczcSURsoc0gcjIiVV9dw9VZaJZMSNljdj45m88QQT1h8jOj6RPo3K8cIDVSjqlduh5/2Xv7fAtIegdk9rag/TH2PYHD1/nfcW7mLY6bdp7nKQCy0/oWSbp5xdlpHJOaoPZnkqi+ggIodFJEREXk/h/YEiEi4ie22PIUneGysiB0UkSES+kWQLSIhIgIgEJnk+J8lxQkVkbyo/m0PkcXflubZVWP9qa/o0KsesHSfxG7uWb/84SlRsBt0MWb4ZtHoDDsyFvbMy5pxGphYVm8DYlcH0GPc7L557k+Yuh6DLeBMuRrpK7TWw3V99bVPKjMdamCwM2CkiAap6KNmmc1R1eLJ9mwHNsVbPBNgE+AHrbO93ByKT7qOqjyXZ/wv+f/2aTKGoV27e71qbgc19GLsymC9+P8LM7X8zsl1VejYo6/gZZ+9/yVpqefnL4N3QuinTyJHWBl9g1OJArl65xJLC4/CJOox0nwR1ejq7NCObSe0VzKRUbNsICFHV46oaC/wKdLFzX8VaqtkdyA24AecBRMQLGAmkOG2w7UrnUWB2KmrNMJWKefFjP1/mPdWUMgU9eW3+ATqN28ja4AuOHXGWywW6TwI3T5g3yJq7zMhRzkVE88wvuxk0bSdFXaPY5v0tFaKDkEemmnAxHCJVAaOq36di8zLAqSTPw2yvJddDRPaLyDwRKWs7z1ZgLXDW9lilqkG27d8HvgButxj9/cB5VT2ailoznK9PYeY/3Ywf+tYnJj6BQdN20ven7QSeduCFV/5S0HUCnA+E1W857jxGphKfkMiUTSdo+8U6/gi6wNttSrAg7yd4XT4Ej/5szcZtGA7gyIHtKbX5JP+KvgTwsc3MvAaYDiAilYEagDdWKLURkZYiUg+orKoL73De3tzh6kVEhorILhHZFR4ebv+ncQARoWOdUqwe4cd7/rUIPnedh7/dxIu/7uHU5dvl5z2q2h6aDoedP8GhAMecw8g09p66Spfxmxmz9BC+PoX5Y1gthoQ8T67ww9B7NlTv5OwSjWwsTXfy23VgkabAaFV90Pb8DQBV/fg227sAl1W1gIi8Anio6vu2994BooHrwCggFqv/qDiwRVVb2bZzBU4DDVQ17G41Zrbp+q9Fx/Hj+mP8tPEEqjCwuQ/Ptqqc/nM+xcfClAfh8jEYttFatMzIVq5Fx/H5qsP8vO1viufLzbuda9GxPMiMLnD1pBUulVo7u0wji3L0nfz22AlUEZEKIuIO9AL+9ZXZNonmLf7ArWawk4CfiLiKiBtWB3+Qqv6gqqVV1QdoARy5FS42DwDB9oRLZpTfw41XHqzOulda4V+vNJM2HqflZ2v5aeNxYuLTccSZqzv0nGLdiDl/CCTEpd+xDadSVQL2naHtF+uZue1vBjbzYc1IPzqVV2T6wxARBo/PM+FiZAiHBYyqxgPDgVVYwTFXVQ+KyBgR8bdt9rxtKPI+4HlgoO31ecAx4ACwD9inqkvsOG0vMmnnfmqUKuDJ54/cx/Ln76de2YJ8sCyItl+sZ/He0yQmptMVZ+EK0PlrCNsBa3P06gvZRujFG/SfsoPnZ++hVAEPAoa34N3OtcgXfRamdoTr56HfAvBp4exSjRzCYU1kWUFmayK7nU1HL/LR8iAOnb1GXe8CvNGxBk0rFUmfgwc8D3/NsH7xVGqTPsc0MlRMfAIT1h1n/LoQcrvk4pUO1ejbuLw19P3yCZjeGaKvQb+F4N3A2eUa2YBDZ1POLrJKwAAkJiqL9p7m81WHORMRTZvqxXm9Y3WqlrjHqV9ib8KkNnDzorXUcj4z91RWsiXkIm8vCuT4xRt0vq80ox6qQfH8Htabl47BtIchPgr6LTJz0RnpxgSMHbJSwNwSHZfAtC2hjF8bwo2YeB71LcuIdlUpceuXSlpcCIKJraFcE3h8AeQys+ZmduHXY/hoeRAL95ymfJE8vN+lNi2rFkuywWHryiUxAfovhpK1nVeske2YgLFDVgyYW67ciOXbP0P4eVsorrly8eT9FRjqVwmv3GmcoHD3NFjyArR9F+4fma61GuknMVGZvfMkn64IJiougaf9KvFM68p4uLn8/0bnD8J0f+vm2v4BULy68wo2siUTMHbIygFzy8lLNxm7Kpil+89S1MudFx6oSq+GZXFL7dodqtYd/ocC4ImVULaRYwo20uzQmWu8tegAe05epWnFIrzftTaViyebnfvsPpjRFVw9YMASKFrZOcUa2ZoJGDtkh4C5Ze+pq3y0PIgdJy5TsVheXutQnfY1SyCpmTk5OgIm3A+aCE9tBE+zNG5mcCMmnq9+P8LULaEU9HTj7Ydr0LVemf/+3YbthpndIHd+GBAAhSs6p2Aj2zMBY4fsFDBg3QPxR9AFPl4RxLHwGzT0KcQbnWqkbg31sN0wpT1U62hNI2Km9ncaVWX1ofOMDjjI2Yho+jQux2sPVk/5xtuT22FmD8hTGAYuhYLlMr5gI8cwAWOH7BYwt8QnJDJn1ym++v0oFyNjeKhOKV55sBo+RfPad4At38Lqt+GhL6DhkLtvb6S7sCs3GR1wkDVBF6heMh8fdqtDg/K3+aIQuhl+eQTylbSaxQqkNOWfYaQfEzB2yK4Bc8uNmHgmbTzOxA3HiUtIpG/j8jzftgqF87rfecfERJj1qDW9/5N/QMk6GVOwQVxCIpM3nWDcmqOIwIgHqjKouQ+ut+tTO74OZvWyrlgGBFghYxgOZgLGDtk9YG65cC2ar9YcZc7Ok+R1d+Xp1pV4onmFf488Su7GRWup5dz5YNh6cLfz6sdIs12hl3lrYSCHz1+nfc0SvOtfizIFPW+/w9E1MKcvFK5kDUX2Knb7bQ0jHZmAsUNOCZhbQi5c55MVh1kTdJ5SBTx4qX01uv2vzO0XOzuxwRruWq8PdE3NSg1Galy5EcunK4P5decpyhT0ZLR/LdrVvMsNr4dXwNz+UKwa9FsMedNpZgfDsIMJGDvktIC5ZdvxS3y8PIh9YRHUKJWfNzpW//dNekn9+SFsGAvdJsJ9j6W8jZEmqsr8v07z0fIgIqLiGNKiAi88UIU87ne5l+nQYpj3BJSsa03xY0b7GRnMBIwdcmrAgPXLben+s4xdFcypy1HcX6Uob3SsQc3S+f+9YUK8dUf4uf0wbAMUqeScgrOZkAvXeWthINtPXKZB+UJ82K021Uvmv/uOB+bBgqHg7Qt9fwOPAo4v1jCSMQFjh5wcMLfExCcwc9tJvv3zKBFRcXT7Xxlebl+N0knb/iPCYEILqyN58O/gmtt5BWdxUbEJfLf2KBM3HCePuytvdKzOo75lyXW7Zsqk9s6Gxc9AuabQZ47VP2YYTmACxg4mYP5fRFQc368LYermUACeaF6BZ1pXIr+H7Z6L4OXwa29o/DR0/MR5hWZh6w5fYNTiQE5djqJHfW/e7FSdIl52hvVfM6yZryu0tBYLM4MuDCcyAWMHEzD/dfpqFF+sOszCvacp6OnGc22q8HiT8ri75oIVr8P2H6CXWWo3Nc5fi2bMkkMsO3CWSsXy8kHXOqlbbmHnT7DsJaj8ADw2E9zuMLLMMDKACRg7mIC5vcDTEXy8IojNIZcoVzgPr3aoxkM1CiNT2ltL7j61CQp4O7vMTC0hUZmxNZQvVh8hLiGR59pUZmjLSlZY22vbD7DydajaER6dbponjUzBBIwdTMDcmaqy4ehFPl4eRPC569xXtiBjWnhw3zJ/awTTgCXgksbZm7O5/WFXeXPhAQJPX8OvajHGdKlF+SKpbNba9DWseRdq+EOPydZS14aRCdgbMGbhD+O2RAS/qsVY9vz9jO1Zl/MR0XSZfY6fCjwHJ7fA+k+dXWKmcy06jncXB9Jl/GYuXIthfJ/6TBvUMPXhsn6sFS61e0DPqSZcjCzJfP007soll/Cob1k61y3NlM0n+HqdKwUT/ei+4TMiSjSmUK0HnF2i090a9j1m6SEuRcYwoKkPL7WvSj6PFCamvPOBYO2HsOEzqNvLusE11x1mXDCMTMyhVzAi0kFEDotIiIi8nsL7A0UkXET22h5Dkrw3VkQOikiQiHwjyeYmF5EAEQlM9tpztvMdFJGxjvtkOZOnuwvPtq7M+ldacbj+O5zQUsTNHcLE5du5ERPv7PKcJvTiDfpP2cFzs/dQMr8Hi55tzmj/WmkLlzXvWuHyv34mXIwsz2FXMCLiAowH2gFhwE4RCVDVQ8k2naOqw5Pt2wxoDtS1vbQJ8APW2d7vDkQm26c10AWoq6oxIlI8fT+RcUsRr9y81c2XsBozKDSnI1W2vkLrv95mRPvqPNLA+/YTM2YzMfEJ/Lj+ON+tDcHdJRfv+dfi8Sblbz/1zp2owso3rFF6voOh0+dm6Wojy3Pk/8GNgBBVPa6qscCvWAFgDwU8AHcgN+AGnAcQES9gJPBBsn2eBj5R1RgAVb1wz5/AuCPv6g1x6/QJrV328YzHKt5YcICO4zbyR9B5svvgkS3HLtJx3Ea+/P0I7WuW4I+X/BjQzCdt4ZKYaA1D3v4DNHnGWibBhIuRDTjy/+IywKkkz8NsryXXQ0T2i8g8ESkLoKpbgbXAWdtjlaoG2bZ/H/gCuJnsOFWB+0Vku4isF5GGKRUlIkNFZJeI7AoPD0/zhzNsfAdDjc4MuDmN2Z3cSEhUBk/fRa+J29gfdtXZ1aW7i5ExjJyzlz6TthOfoEx/ohHf9alPifweaTtgYgIseR52TYbmL8CDH5lF3oxsw5EBk9K/kuRfa5cAPqpaF1gDTAcQkcpADcAbK5TaiEhLEakHVFbVhSkc2xUoBDQBXgHmJu+3AVDViarqq6q+xYqZ6c3vmQj4f4vkK03Tv15m1dP38X6XWoRciMT/u808N3sPpy4n/y6Q9SQmKrO2n6TtF+tZsv8Mz7WpzOoRLfG73SSh9kiIh0XPwJ6foeWr8MB7JlyMbMWRo8jCgLJJnnsDZ5JuoKqXkjydBNwa99oN2KaqkQAisgIrOK4DDUQkFKv24iKyTlVb2c63QK22mR0ikggUBcxliqN5FoKek2FKB9yWj6Bfz6l0/V8ZJm44zqSNx1kZeJb+TX14rk1lCubJesNtg85e462FB/jr5FWaVCzMB13rULm4170dNCEOFg6DwPnQ+m3weyV9ijWMTMSRVzA7gSoiUkFE3IFeQEDSDUSkVJKn/sCtZrCTgJ+IuIqIG1YHf5Cq/qCqpVXVB2gBHLGFC8AioI3tuFWx+m8uOuSTGf9VthG0eRsOLoS/ppPPw42X2ldj3cut6f4/b6ZuPkHLsWv5cf0xouMSnF2tXW7ExPPR8iAe/nYTf1+6yZeP3sfsJ5vce7jEx8K8QVa4tBtjwsXIthx6J7+IdAK+BlyAKar6oYiMAXapaoCIfIwVLPHAZeBpVQ22jUD7HmiJ1ay2UlVHJju2D7BUVWvbnrsDU4B6QCzwsqr+eaf6zJ386SwxEWZ2h5NbYeg6KF7jn7cOn7vOJyuCWHs4nDIFPXn5wap0ua+MfbMIO8Hqg+cYHXCQMxHR9G5Ujtc6VEufq6/4GJg7AI6sgA6fQJOn7/2YhpHBzFQxdjAB4wCRF6yllvMUhifXgnuef729JeQiH60IIvD0NWqVzs+bnWrQvHJRJxX7X2FXbjI64BBrgs5TvWQ+PuxWmwblC6fPweOi4Ne+cOwPa6RYwyF338cwMiETMHYwAeMgx/6En7tD/f7g/81/3k5MVJbsP8PYlYc5fTWKVtWK8XrH6vYtuOUgcQmJTNl0gq/XHAVgRLsqDGpeAbf0uqcn9gbM7gUnNlp/JvX7p89xDcMJTMDYwQSMA615DzZ9CT2nWPNppSA6LoGft/7Nt38eJTImnp4NvBnZrholC6RxyG8a7f77Mm8tDCT43HXa1SzBaP9alCmYjlPix1yHWY9ZTYddf4D7eqXfsQ3DCUzA2MEEjAMlxMHUThAebC21XLjCbTe9ejOW8WtDmL7lb3LlgiEtKjLMr2Lqp1pJpas3Y/l0ZTCzd5yidAEPRvvXon2tkul7kugImNkTTu+GHpNuG7aGkZWYgLGDCRgHu3rSWmq5cCV4YtVdZwQ+dfkmn68+zOK9ZyiS150XHqhC70bl0q+ZykZVWfDXaT5aHsTVqDiGtKjA822rkDd3Oo/aj7piNRWe22/NiFzTP32PbxhOYgLGDiZgMsChAJjbD5oOhwc/tGuX/WFX+Wh5ENuOX6ZC0by81qEaD9YqSQr3zaZayIVI3l50gG3HL1O/XEE+7FaHGqUc0Pdz4xL83AXCD8OjM6Bax/Q/h2E4iQkYO5iAySDLXrKW/e3zG1Rtb9cuqsrawxf4eHkwRy9E0qB8Id7sVD3NI7qi4xIYvzaECeuPkcfdldc7Vucx37KOGSYdGQ4z/OHycXjsF6hiljMwshcTMHYwAZNB4qLhp7Zw/Sw8tRnyl7r7PjbxCYnM2x3Gl78f4cL1GDrUKslrHatToaj9C3itPxLOqEWBnLx8k+71y/BmpxoU9XLQ0sPXz8F0f6t5sM+vULGVY85jGE5kAsYOJmAyUPgRmOgHZRpA/8WpXufkZmw8P208wY/rjxETn0ifxuV4vm2VOwbF+WvRjFl6iGX7z1KxWF4+6FqbZpUceM9NxGmY3hkiz0OfueDT3HHnMgwnMgFjBxMwGWzvLFj0NLR+C/xeTdMhwq/HMO6PI8zecQpPNxee8qvI4BYV8XT//8BKSFRmbvubz1cdJjYhkeGtKzPUryK5XR24eNeVv61wiboCfedBucaOO5dhOJkJGDuYgMlgqtYEjwd+gwFL7+kb/rHwSD5dEczqQ+cpkT83L7WrRo8G3hw6c403Fx7gwOkI7q9SlA+61qZ8Efub09Lk8nGrWSzmGvRbaF2lGUY2ZgLGDiZgnCDmOvzoZ02b8vRma0qZe7Az9DIfLQ9iz8mrlC3syekrURTxys27nWvyUJ1S6TLy7I4uHrWuXOJjoP8iKHWfY89nGJmACRg7mIBxkjN7YXI7qNQWes++5zVQVJUVgeeYvOkEdcoUYGT7quR38E2aAFwItsIFtfqVStRy/DkNIxOwN2AcuR6MYaSsdD1o9z6sfA22T7jnGYVFhE51StGpjv2j0+7ZuUCY0QVyucKAAChWLePObRhZhFn423COxsOgWidYPQrO7HF2NalzZi9Mfxhcc8Og5SZcDOM2TMAYziECXcaDV3GY94TVN5MVhO2yOvTd88HAZVCkkrMrMoxMywSM4Tx5CkOPn+BKKCwdaY0yy8xOboMZXSFPIRi07I4TeBqGYQLGcLbyzaDVm3BgrnWfTGZ1YqM1cWW+EjBoBRQs5+yKDCPTMwFjON/9I8Hnflj+sjU5ZGZzbC388ggULAsDl0P+0s6uyDCyBIcGjIh0EJHDIhIiIq+n8P5AEQkXkb22x5Ak740VkYMiEiQi30iyGxpEJEBEApM8Hy0ip5Mcq5MjP5uRjnK5QPdJ4JbH6o+Ji3J2Rf/vyGprsbDCFa2bQ/OVcHZFhpFlOCxgRMQFGA90BGoCvUWkZgqbzlHVerbHT7Z9mwHNgbpAbaAh4Jfk2N2ByBSO9VWSYy1P309kOFT+UtBtApwPhNVvO7saS/Ay+LUPFK8OA5eCVzFnV2QYWYojr2AaASGqelxVY4FfgS527quAB+AO5AbcgPMAIuIFjAQ+SPeKDeeq0g6aPWdN7X9osXNrObgI5vaHUnWhf8A9zzhgGDmRIwOmDHAqyfMw22vJ9RCR/SIyT0TKAqjqVmAtcNb2WKWqQbbt3we+AG6mcKzhtmNNEZFC6fVBjAzU5h1rLq/Fz1kTSDrD/t+sproyvtBvEXgWdE4dhpHFOTJgUpr/I/k41CWAj6rWBdYA0wFEpDJQA/DGCqU2ItJSROoBlVV1YQrH/gGoBNTDCqUvUixKZKiI7BKRXeHh4Wn4WIZDubpDj8mAwvzBkBCXseffOwsWPAnlmsLj88HDAatdGkYO4ciACQPKJnnuDZxJuoGqXlLVGNvTScCtaWi7AdtUNVJVI4EVQBOgKdBAREKBTUBVEVlnO9Z5VU1Q1UTbsRqlVJSqTlRVX1X1LVbMtKlnSoUrQOdxELYT1tq3zHK62D0NFj0DFf2g72+Q2yvjzm0Y2ZAjA2YnUEVEKoiIO9ALCEi6gYgknTzKH7jVDHYS8BMRVxFxw+rgD1LVH1S1tKr6AC2AI6raKoVjdeP/2rvzaKvKOozj3weQyCwyyZY5hCYgaho4ZA4Y5gROOZRDarDUBnNIK9RcEUVpWoar0hTNRlMTJzCVZSolyHVmELEkJSMnLCs1Uy/8+uN90cP1Dgc4e5995fmsdRdneM/Zz4G7+Z09/V54COu+tjwIthkF0yfAgtuKX949l8CUk2HT3eHwq6D3msUv0+wtrrACExGtwAnAVFLh+G1EzJP0LUn752En5VORZwMnAaPy45OAvwBzgdnA7IiY0sUiz5U0V9IcYDhwSmM/kZVur7PhvYPTHDIvPFPccmZekK7BGbQPHHY5rNGnuGWZrUbcrt/t+qvt2fkwcXiaIfLI66BHg78T3fkDuO2bsPkB6dhPzxLa/Jt1c/W26/eV/FZt6w6GEefAY9NgxoTGvW8ETDsnFZcPfRIOvszFxazBXGCs+oYeDVscBLd/B564e9XfLwJuHw/TzoKtj4ADL4aens72aHMAAApYSURBVBrJrNFcYKz6JNjv/NQL7Jpj4OXnV/69IuDWr8Od58HQz6QpA3r0bFxWM3udC4x1D336wiGXwQtPww0nrFxr/wi45XS460ew3XGw7/mNP6ZjZq/z2mXdx/rbwO7j4JEbUzuZFbF0Kdx4Sp6i+Ysw8nsuLmYF8xpm3csOx8OAPWHqmfD03Ppes3QJTD4R7v8Z7HwK7PWdtNvNzArlAmPdS48e8ImfpOaTV4+GV9prql1jSStc93mY9WvY9XT4+DdcXMxK4gJj3c87+qX5Y/6xAG4e0/G4Ja/Btcem2TJ3+zoMP8PFxaxELjDWPW28C+w6BmZdDrOvevPzra/C1aNg3nWw57dh2FdKj2i2unOBse5r2Bj4wE7p4P1zC954/LX/wVVHppMBRpyb5pgxs9K5wFj31bNX2lXWqzdMGg2tr8Cr/4UrD4dHp8K+E+Ajn2t2SrPVli9ftu6t7/rpoP8Vh8HNp6XjMgunpwsohxzZ7HRmqzUXGOv+Bo1Ipy+3XAjqkVq/bH1os1OZrfZcYOytYfdx8OpLMGAPGLxfs9OYGS4w9lbR622w/w+bncLMavggv5mZFcIFxszMCuECY2ZmhSi0wEjaW9KfJC2QdHo7z4+StFjSrPxzbM1z50qaJ2m+pB9Ky/f4kDRZ0kPtvOdXJIWkfsV8KjMzq0dhB/kl9QQuAPYAFgH3SpocEQ+3GXpVRJzQ5rU7AjsBW+WHpgO7AtPy8wcBb+pyKGnDvLwnGvdJzMxsZRS5BbM9sCAiHouIV4ErgQPqfG0AfYDewNuANYBnACStBZwKfLud100AxuTXm5lZExVZYNYH/lZzf1F+rK2DJc2RNClvgRARM4E7gKfyz9SImJ/HjwfOA/5b+yaS9gf+HhGzG/sxzMxsZRRZYNrri952y2IK0D8itgJ+D/wCQNKmwGBgA1JR2k3SMEkfBjaNiOuWW5C0JnAmMLbLUNJnJd0n6b7Fixev6GcyM7M6FXmh5SJgw5r7GwBP1g6IiH/U3L0EOCffPhBoiYgXASTdDOwAvABsI2khKfu6kqYBJwIbA7PzuQAbAA9I2j4inm6zzInAxPy+iyX9dZU/aeP0A55rdohOVD0fVD9j1fNB9TNWPR+89TN+oK5REVHID6kAPEb6j783MBvYos2Y9WpuLysqAIeStmh6kY6/3Abs1+a1/YGHOlj2QqBfUZ+twL+z+5qdoTvn6w4Zq56vO2Ssej5nfOOnsC2YiGiVdAIwFegJXBYR8yR9K3+wycBJ+dhJK/BPYFR++SRgN2AuabfaLRExpaisZmbWeIX2IouIm4Cb2jw2tub2GcAZ7bxuCdDpRB4RsRDYsoPn+q94WjMzayRfyV8tE5sdoAtVzwfVz1j1fFD9jFXPB84IgPK+ODMzs4byFoyZmRXCBaZkXfVny2M+Jenh3IvtN1XLKGkjSXdIejBfJDuy5HyXSXq2vV50+Xnl/nULcr6hFcv36ZxrjqS7JG1dZr56MtaM207SEkmHlJWtZtldZpT0sdzHcJ6kP1Qpn6S+kqZImp3zjS4534Z5PZ2fl39yO2OKXVeafarc6vRDOpvuL8AmvHHq9uZtxgwAHgTWzvfXrWDGicAX8u3NgYUlZxwGDKXj09RHAjeTLvbdAbi7Yvl2rPn3HVF2vnoy1vwu3E46UeeQqmUE3g08DGyU75e9rnSV72vAOfn2e0lnyvYuMd96wNB8+53An9tZlwtdV7wFU656+rMdB1wQEc8DRMSzFcwYwLvy7b60uYC2aBHxR9LK2pEDgF9G0gK8W9J65aTrOl9E3LXs3xdoIV0YXKo6/g4hXcB8DVD27yBQV8YjgGsj4ok8vtScdeQL4J25E/xaeWxrGdkAIuKpiHgg334BmM+b23UVuq64wJSrnv5sA4GBkmZIapG0d2npknoyjgOOlLSI9O32xHKi1a3ePnhVcAzpG2SlSFqfdPHzRc3O0omBwNqSpkm6X9LRzQ7Uxo9JLa+eJF3Td3JELG1GEEn9gSHA3W2eKnRdKfQ6GHuTevqz9SLtJvsY6ZvtnZK2jIh/FZxtmXoyHg78PCLOk/RR4Fc5Y1NWnnbU8xmaTtJwUoHZudlZ2nE+cFpELGkzFVOV9AK2AT4OvB2YKaklIv7c3Fiv2wuYRbpo/IPArZLujIj/lBkid6C/BvhSO8sudF1xgSlXl/3Z8piWiHgNeFzSn0gF595yItaV8Rhgb0idryX1IfU1asqulHbU8xmaStJWwKXAiFi+J19VbAtcmYtLP2CkpNaIuL65sZazCHguIl4CXpL0R2Br0rGGKhgNfDfSwY4Fkh4HNgPuKSuApDVIxeXyiLi2nSGFriveRVaue4EBkjaW1Bs4DJjcZsz1wHAApVk5B5J6ulUp4xOkb41IGkyau6dKraknA0fnM2R2AP4dEU81O9QykjYCrgWOqtC37eVExMYR0T9SV4xJwPEVKy4ANwC7SOqVO6p/hHScoSpq15P3AYMocV3Ox35+CsyPiB90MKzQdcVbMCWK+vqzTQX2lPQwsAT4apnfcOvM+GXgEkmnkDanR+VvaaWQdAVpF2K/fBzoG6SmqETERaTjQiOBBaR5g8o+PbSrfGOBdYAL8xZCa0RsW7GMTddVxoiYL+kWYA6wFLg0Ijo97brMfKS5q34uaS5pV9RpEVFmh+WdgKOAuZJm5ce+BmxUk7HQdcVX8puZWSG8i8zMzArhAmNmZoVwgTEzs0K4wJiZWSFcYMzMrBAuMGZmVggXGLNuILelv3FVx5iVyQXGrAHyldBen8xqeIUwW0mS+ufJnC4EHgCOkjRT0gOSrs5NBpE0UtIjkqbnyZ063MqQtL3SJGQP5j8HtTNmnKRfSbpd0qOSjqt5ei1Jk/LyLs/tQpA0VtK9kh6SNHHZ42ZFcoExWzWDgF8Ce5CagO4eEUOB+4BTcyPQi0lNLXcmTTzVmUeAYRExhNRS5qwOxm0F7AN8FBgr6f358SHAl0gTwW1CahcC8OOI2C4itiR1Ht53hT+p2QpyLzKzVfPXiGiRtC/pP/UZeeOgNzCT1D33sYh4PI+/AvhsJ+/XF/iFpAGkPm9rdDDuhoh4GXhZ0h2kieL+BdwTEYsAcv+p/sB0YLikMcCawHuAecCUlfvIZvVxgTFbNS/lPwXcGhGH1z4pacgKvt944I6IODBPEjWtg3Ftmwguu/9KzWNLgF55K+pCYNuI+JukcaQO2GaF8i4ys8ZoAXaStCmApDUlDSTt8tokFwuAQ7t4n77A3/PtUZ2MO0BSH0nrkDr6djZf0LJi8lw+LnRIFxnMGsIFxqwBImIxqSBcIWkOqeBslndjHQ/cImk68Azw707e6lzgbEkzSNMldOQe4Hd5OeMjosNJovJsqJeQpu29nvImr7PVnNv1mxVM0loR8WI+c+sC4NGImLAK7zcOeDEivt+ojGZF8BaMWfGOywfc55F2gV3c5DxmpfAWjFkTSBoNnNzm4RkR8cVm5DErgguMmZkVwrvIzMysEC4wZmZWCBcYMzMrhAuMmZkVwgXGzMwK8X+2dDnod+1GKAAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x1659130a320>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# summarize results\n",
    "print(\"Best: %f using %s\" % (gsearch5_1.best_score_, gsearch5_1.best_params_))\n",
    "test_means = gsearch5_1.cv_results_[ 'mean_test_score' ]\n",
    "test_stds = gsearch5_1.cv_results_[ 'std_test_score' ]\n",
    "train_means = gsearch5_1.cv_results_[ 'mean_train_score' ]\n",
    "train_stds = gsearch5_1.cv_results_[ 'std_train_score' ]\n",
    "\n",
    "pd.DataFrame(gsearch5_1.cv_results_).to_csv('my_preds_reg_alpha_reg_lambda_1.csv')\n",
    "\n",
    "# plot results\n",
    "test_scores = np.array(test_means).reshape(len(reg_alpha), len(reg_lambda))\n",
    "train_scores = np.array(train_means).reshape(len(reg_alpha), len(reg_lambda))\n",
    "\n",
    "#log_reg_alpha = [0,0,0,0]\n",
    "#for index in range(len(reg_alpha)):\n",
    "#   log_reg_alpha[index] = math.log10(reg_alpha[index])\n",
    "    \n",
    "for i, value in enumerate(reg_alpha):\n",
    "    pyplot.plot(reg_lambda, -test_scores[i], label= 'reg_alpha:'   + str(value))\n",
    "#for i, value in enumerate(min_child_weight):\n",
    "#    pyplot.plot(max_depth, train_scores[i], label= 'train_min_child_weight:'   + str(value))\n",
    "    \n",
    "pyplot.legend()\n",
    "pyplot.xlabel( 'reg_alpha' )                                                                                                      \n",
    "pyplot.ylabel( '-Log Loss' )\n",
    "pyplot.savefig( 'reg_alpha_vs_reg_lambda1.png' )"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    " -0.584433 using {'reg_alpha': 2, 'reg_lambda': 1} 增加正则后效果是有点提升，但是不大。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "得到最优的参数结果如下：\n",
    "        learning_rate =0.1,\n",
    "        n_estimators=340,  #第二轮参数调整得到的n_estimators最优值\n",
    "        max_depth=4,\n",
    "        min_child_weight=1,\n",
    "        gamma=0,\n",
    "        subsample=0.7,\n",
    "        colsample_bytree=0.8,\n",
    "        colsample_bylevel = 0.7,\n",
    "        objective= 'multi:softprob',\n",
    "        reg_alpha=2,\n",
    "        reg_lambda=1,\n",
    "        seed=3)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 划分数据，把模型的参数存入，进行训练，得到准确率"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 161,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "D:\\anaconda\\lib\\site-packages\\sklearn\\model_selection\\_split.py:2026: FutureWarning: From version 0.21, test_size will always complement train_size unless both are specified.\n",
      "  FutureWarning)\n"
     ]
    }
   ],
   "source": [
    "from sklearn.model_selection import  train_test_split\n",
    "X_train1,X_test,y_train1,y_test = train_test_split(X_train,y_train,train_size= 0.2,random_state = 666)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 168,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Accuracy: 72.73%\n"
     ]
    }
   ],
   "source": [
    "model = XGBClassifier(\n",
    "        learning_rate =0.1,\n",
    "        n_estimators=340,  #第二轮参数调整得到的n_estimators最优值\n",
    "        max_depth=4,\n",
    "        min_child_weight=1,\n",
    "        gamma=0,\n",
    "        subsample=0.7,\n",
    "        colsample_bytree=0.8,\n",
    "        colsample_bylevel = 0.7,\n",
    "        objective= 'multi:softprob',\n",
    "        reg_alpha=2,\n",
    "        reg_lambda=1,\n",
    "        seed=3)\n",
    "model.fit(X_train1, y_train1)\n",
    "ans=model.predict(X_test)\n",
    "predictions = [round(value) for value in ans]\n",
    "accuracy = accuracy_score(y_test, predictions)\n",
    "print(\"Accuracy: %.2f%%\" % (accuracy * 100.0))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "使用训练好的模型在测试集进行预测。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 163,
   "metadata": {},
   "outputs": [],
   "source": [
    "y_pred1 = model.predict(test)\n",
    "pred1 = [round(value) for value in y_pred1]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "保存预测结果"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 164,
   "metadata": {},
   "outputs": [],
   "source": [
    "result = pd.Series(pred1, name='interest_level')\n",
    "target_num_map = {0:'high', 1:'medium', 2:'low'}\n",
    "result = result.map(target_num_map)\n",
    "pd.DataFrame(result).to_csv('pred_result.csv',index_label ='ID')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 作业总结\n",
    "\n",
    "\n",
    "### 1、仅仅靠参数的调整和模型的小幅优化，想要让模型的表现有个大幅度提升是不可能的。加入正则调优确实是有一定的提升，但是没有达到质的飞跃。\n",
    "\n",
    "### 2、要想让模型的表现有一个质的飞跃，需要依靠其他的手段，诸如，特征工程(feature egineering) ，模型组合(ensemble of model),以及堆叠(stacking)等。\n",
    "\n",
    "### 3、模型训练速度上非常慢，电脑配置还凑合，基本跑10几个小时，是属于正常的，主要是使用了Sklearn的XGBoost，搜索发现GPU加速还有同学直接使用XGBoost 速度提升很多，后续可考虑学习如何使用减少训练时间。都是通过使用GPU加速实现的。\n"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}
